Template versioning requirements in CDRs

[ian.mcnicoll]

As we transition to ADL2, and update the REST API, there are a couple of issues that are arising with template-version handling in CDRs.

What is changing…

• templateIds will become fully versioned, namespaces, 'technical identifiers e.g. org.openehr::openEHR-EHR-COMPOSITION.vital_signs_encounter.v1.2.34 . In transition existing uncontrolled templateIds e.g. UCP - Vital signs will still be allowed and retained in ADL2 templates as a ‘title’ for human visualisation in tooling.

We expect the new structured templateIds will be automatically generated in a similar
that archetypeIds are generated.

• There are discussions in the REST API group about how templates should be handled in CDRs, in particular, which template updates into the CDR should be allowed without special admin rights, or indeed whether some updates should be prevented altogether e.g. a non-breaking update to an existing template where the current templates is in-use by one or more compositions in production

The good aspect of these changes is that we will be able to exert much more control over the revisions of templates that are deployed, improving clinical safety and data coherence.

The downside is that if not managed carefully, it will present a significant burden to modellers and developers, especially in pre-deployment situations where frequent changes to a template are common.

Different CDRs currently behave somewhat differently.

The BetterCDR allows templates to be overridden, as long as the template name does not change. EHRBase does allow updates, but only via an Admin API, and only if the template has never been used by compositions.

As we move into a situation where we are using ADL2 ‘archetypeHRIDs’ and every revision to a template creates a new templateId string, we need to consider the appropriate behaviour.

As an example

1. We create a new template com.freshehr::openEHR-EHR-COMPOSITION.vital_signs_hexham_icu.v1.0.0

2. After 6 months, the client asks for a currently constrained node to be made visible, perhaps Mean Arterial BP in a BP archetype. This is a non-breaking change, so the new revision is com.freshehr::openEHR-EHR-COMPOSITION.vital_signs_hexham_icu.v1.1.0.

Currently both BetterCDR and Ehrbase would treat this as a completely new template, but really it is a minor revision of an existing template. I would assume that we would always want any compositions to be validated against the latest revision? Do we need to keep the previous revisions (I assume, yes), but only normally display / return the most recent revision. i.e do we ask the CDRs to treat them internally as revisions of the same template, not as new templates.

A major revision e.g. com.freshehr::openEHR-EHR-COMPOSITION.vital_signs_hexham_icu.v2.0.0 would be treated as if it were a new template.

An obvious downstream impact will be on other tooling such as form builders that may not want to be tied to a specific minor revision, so need to reference com.freshehr::openEHR-EHR-COMPOSITION.vital_signs_hexham_icu.v1 (which ADL2 calls an archetype_reference) and not the full archetypeHRID . i.e any revision or patch change below v1 is acceptable. The impact here is really on downstream tooling.

Finally, we also need to consider the impact on pre-publication templates.

We now use com.freshehr::openEHR-EHR-COMPOSITION.vital_signs_hexham_icu.v1.0.0-alpha.1 as our starting revision for new templates (and indeed local archetypes), as this makes transition into publication easier, since we don’t have to switch from .v0 to .v1 when we deploy, disrupting all sorts of downstream activity.

Do the rules need to be any different for pre-publication templates?

Possibly the safest approach is to be able to explicitly mark a CDR or CDR domain as being ‘development’, so that looser rules around updates and deletions can be applied. Ehrbase has something like this in config but it is a bit obscure and is still overly restrictive in some places.

[joostholslag]

@ian.mcnicoll

There are discussions in the REST API group about how templates should be handled in CDRs, in particular, which template updates into the CDR should be allowed without special admin rights, or indeed whether some updates should be prevented altogether e.g. a non-breaking update to an existing template where the current templates is in-use by one or more compositions in production

I would say, the REST API shouldn’t be concerned with authorization. Setting special admin rights is up to the CDR configuration, not the REST api (endpoint). I do think it’s key to align the api with the SMART on openEHR scopes. So that it’s possible to authorize a scope (e.g. templateId).

(So I also don’t like the new admin APIs with only delete operations for resources in other APIs)

I would love to standardize how to do authorization in openEHR. But that’s quite a big challenge. And I would really like to keep those discussions seperate.

Currently both BetterCDR and Ehrbase would treat this as a completely new template, but really it is a minor revision of an existing template. I would assume that we would always want any compositions to be validated against the latest revision? Do we need to keep the previous revisions (I assume, yes), but only normally display / return the most recent revision. i.e do we ask the CDRs to treat them internally as revisions of the same template, not as new templates.

Yes, CDRs need to handle those as minor revisions of an existing template, not new ones.
But no, don’t assume any VERSION to validate against the latest revision. A composition must validate against the (operational)template it has in LOCATABLE.node_id and archetype_details at time of commit. It should reference the used template conclusively, so a unique version of an operational template. Since those should be immutable, logically it will always validate. But they don’t need to validate against new versions of a template always. If you consider only minor version updates the same template, logically some LOCATABLE will always validate against new minor versions of its template. But I think it helps to be strict in the rules: so only validate at commit.

The BetterCDR allows templates to be overridden, as long as the template name does not change. EHRBase does allow updates, but only via an Admin API, and only if the template has never been used by compositions.

So a new minor version of a template should not override an existing template. It should be a new version of the same template. New compositions will (probably) record the new version in their node_id/archeypte_details. But older ones don’t update automatically.
Now I can see some edge cases/migrations where it’s useful to overwrite a specific version of a template with a new template. But since that’s very unsafe, I think we should keep that out of the specification. And specify templates are immutable, just like archetypes and compositions.

We now use com.freshehr::openEHR-EHR-COMPOSITION.vital_signs_hexham_icu.v1.0.0-alpha.1 as our starting revision for new templates (and indeed local archetypes), as this makes transition into publication easier, since we don’t have to switch from .v0 to .v1 when we deploy, disrupting all sorts of downstream activity.

Do the rules need to be any different for pre-publication templates?

Possibly the safest approach is to be able to explicitly mark a CDR or CDR domain as being ‘development’, so that looser rules around updates and deletions can be applied. Ehrbase has something like this in config but it is a bit obscure and is still overly restrictive in some places.

Using .v1.0.0-alpha.n makes sense for the reason you specify, for templates too. But using .v0 should also be fine. But it’s a bit arbitrary since there’s pros and cons for both. I would say it’s project specific. And I think the rules are well defined. The rules should be the same between templates and archetypes, since templates are archetypes.

I agree a CDR deployment should be configurable to allow ‘-alpha’ archetypes and templates, probably this makes sense to allow only on non-production deployments. This is also what Nedap does (and one reason they want to used published CKM archetypes only.)

@ian.mcnicoll

Do we need to keep the previous revisions (I assume, yes), but only normally display / return the most recent revision.

Yes, keep them always! About display: you get the template that is requested. Usually I would say this is the full HRID. So if you want to know the schema for a VERSION you get the template that’s in its node_id, with a full version number there. Now I could see it’s useful to do a get with a partial match e.g. as you described with forms: GET templateX.V1 getting the latest minor version, e.g. for an (empty) form that specifies only a major version. (Similar to the logic for references in specialisation).

A major revision e.g. com.freshehr::openEHR-EHR-COMPOSITION.vital_signs_hexham_icu.v2.0.0 would be treated as if it were a new template.

We should be careful with calling a major version with the same template concept_id a new template. Computationally it’s (mostly) true. But semantically and functionally it’s very useful to regard a major version of a template as a new major of the same template. This discussion is going elsewhere, so let’s park this for now: Delete explanation of major version in HRID in master04-versioning.adoc by joostholslag · Pull Request #22 · openEHR/specifications-AM · GitHub

[ian.mcnicoll]

I’m pretty well much in agreement.

Does this work better

A major revision e.g. com.freshehr::openEHR-EHR-COMPOSITION.vital_signs_hexham_icu.v2.0.0 would need to be treated technically as if it were a new template, since there can be no guarantee that any or all data paths remain compatible, even though it is the ‘same template clinically’

@joostholslag

A composition must validate against the (operational)template it has in LOCATABLE.node_id and archetype_details at time of commit.

Yes, agreed but I think it is an open question about what should happen when a composition needs to be updated - does it re-commit against the existing revision in template_id or against the most recent revision in the CDR?

I think an easier default position should be that a re-commit should normally be against the latest_revision - this will be the correct approach for persistent compositions, and even for older event compositions, as long as the changes are only minor or patch, the re-commit should validate safely. There may be other medico-legal arguments when correcting faulty old data where a re-commit against the original revision might be required, but I;d see these as exceptions.

But I think it helps to be strict in the rules: so only validate at commit.
Agree

[joostholslag]

@ian.mcnicoll

Yes, agreed but I think it is an open question about what should happen when a composition needs to be updated - does it re-commit against the existing revision in template_id or against the most recent revision in the CDR?

I would say that’s (client) implementation choice. If you re-commit against a newer version of that template you must update the template_id (version). If you re-commit against the older version, you must ensure it’s still correct.

I think the context of the use case differs too much to set a default. (Although there’s something to say for picking a default always.)

[pablo (Pablo Pazos)]

Hi,

I disagree in treating a new major version on a template as a new template. Having incompatible paths is part of what makes a template to increase the major version in the semver, but semantically it’s the same template just with a potential backwards incompatible change applied. Something like that happens with archetypes too, it’s not clear why a major version change needs a different treatment for templates.

Another technical thing is that we know for sure which paths changed, since we can do a semantic diff between the templates. In fact, I have implemented such thing in the openEHR Toolkit to diff OPTs 1.4 (I think the diff can be improved by applying AI to understand semantic similarities e.g. when a node name or description changes). Knowing the diff, the system can automatically check stuff like which queries depend on a path that was modified and report that back so updates can be done.

@ian.mcnicoll

I think an easier default position should be that a re-commit should normally be against the latest_revision - this will be the correct approach for persistent compositions

My understanding is that a COMPOSITION should be compatible with the exact template version that was originally used to create the COMPO in the first place. For a re-commit, a human-based data migration would be needed since someone should be responsible in case any data goes missing. For instance, a path that exists in v1 might not exist in v2, so the COMPO can’t be committed for v2 without losing that data in the last version of the COMPO, but the data for that path will be available in the previous version of the COMPO (note re-commit will also create new versions for the COMPO for a new version of the template). So the re-commit is technically possible, but might have some medico-legal considerations/implications, like someone being responsible for that “migration”.