OPT Versioning on CDRs, esp. EHRbase

Implementation Platform
, ,
1

We are investigating whether CDRs support versioning or auditing for OPTs, for example who uploaded or replaced a template and when; similar to how VERSIONED_OBJECT works for compositions.

This seems relevant when a template is overwritten and the new version causes problems with existing compositions, requiring access to the previous version. It could also be important for troubleshooting, auditing, and production data governance.

I could not find anything about OPT versioning in the openEHR specifications.

Better provides a proprietary mechanism: overwritten OPTs are “retired”, only the latest version remains active, and an /admin/rest/v1/templates API supports retiring and unretiring templates. It also exposes basic version data such as upload and retirement timestamps and the OPT XML.

EHRbase provides an administrative template API at /rest/admin/template, according to its documentation, but as far as I can tell it supports updating and deleting templates without keeping a version or audit history.

Without a common CDR-level mechanism, applications need to implement their own template governance and versioning service. Some application-level governance is likely necessary anyway, but basic template versioning still seems like a core CDR capability that applications should be able to rely on, particularly as a fallback when something goes wrong in the application or when templates are modified by third-party actors.

My questions are:

  1. Does EHRbase or HIP provide an OPT versioning or audit mechanism that I have overlooked?
  2. Is OPT versioning intentionally out of scope for the openEHR specifications, or is it simply a gap that has not yet been addressed?
2

Hi,

These are good questions and valid concerns - thanks you.

At Cadasto we also support a similar mechanism to Better, each new POSTed OPT retires the previously available version, and operations are log-audited.

Given your interest on this aspects, I would suggest you to help us identifying the necessary behavior so that is not remains “out of scope” on the specification side. I can infer a bit from your questions how do you expect this feature to work, but if you want to contribute a bit, please make a SPECPR Jira ticket on ITS-REST component (see this Jira list) and we will try to discuss it in the WG.

3

In Atomik and the openEHR Toolkit, there is OPT versioning using semver. When there is a collision, we allow the user to pick what to do and how to modify versioning, if it’s a patch, minor or major change.

We also manage an active/inactive status for OPTs in parallel to the versioning.

We are not using VERSION/AUDIT_DETAILS for this, but we could.

Still we fill that’s not sufficient, since managers should be extra careful in managing the OPTs, delegating to them all to research what kinds of problems each OPT update could cause in a running system.

That’s why we are working on something to solve that problem, hopefully next year we can have something to share. It’s basically a semantic change analyzer, with traceability and migration planning (automating the steps needed for data and query migration when an OPT is updated in different ways).

We are open for collaboration with anyone interested in the topic.

4

Good questions!!

I think the changes being suggested to (optionally) align ADL1.4 / opt templateIds to the new ADL2.4 HRID (fully namespaced/semverioned ID should help here, along with proposed changes to the REST API.

If use of the HRID form of templateId is fully supported /encouraged by tooling, it will mean that every patch change will generate a new templateId and if these are uploaded ot CDR, they should sit alongside the previous version.

Now that raises a problem since we normally only want o use the most recent version, and it adds a burden
for down-stream tooling that relies on the templateId e.g Form builders, AQL tooling, mapping etc, which is particularly problematic during development. And this is where Archetype References can support partial matching which e.g can ignore patch number or even revision number.

I don’t think VERSION/AUDIT DETAILS is quite right for this. @pablo _ agree we need something to support this kind of complex changing environment, certainly when commiting new .opts but ideally as a ‘test run’ in advance, so as a clinical modeller I can test out the impact of updated archetypes and templates before committing the .opts.

5

That’s the key: how to do the test run, and which processed are involved.

In one end you have total manual work: manual tracking of differences between OPTs, manual gathering of queries and sample data, manual copy of production data into a test environment, update OPTs and writing/executing test manually. Then you need to fix all the queries affected by the changes, design an implement data migrations manually, test that, and finally run it in prod. All that can be done today, but it’s painful and prone to errors on everything that says “manual”.

What we are working on is on automating all those steps. From change tracking and semantic diff, to automating data migration plans and auto-generating tests. I’ll publish something here when the first prototype is available.

6

Love this topic.

I also think this is a genuine gap rather than something that’s been overlooked in the specifications.

My view is fairly simple.

First, a CDR should retain immutable OPT versions. Once an OPT has been used to validate committed data, it shouldn’t be overwritten. Uploading a new OPT should create a new version while keeping the previous versions available for audit, troubleshooting and rollback.

Second, I don’t think the CDR should author OPT versions. Versioning, whether that’s semantic versioning or something else, belongs to the modelling and governance process. The CDR’s responsibility is to store those versions and make them available.

Third, I see OPTs being used in two different scenarios, and I think version resolution should reflect that.

When committing a composition, the CDR should validate against the latest active OPT by default, unless the client explicitly requests a specific version.

When reading a composition, the CDR should always use the exact OPT version recorded in that composition so the data can be interpreted, queried and rendered in the same way it was when it was originally committed.

To me, that’s the minimum behaviour a CDR should provide. Everything else, such as semantic diff, impact analysis, migration planning and dependency tracing, builds on having that immutable template history in the first place.

7

@HHeiser Since version v2.29.0 EHRbase supports template versioning better. The support of versioning is more implicit rather than explicit.

For the mechanism to work, the template names need to follow a specific format {template_name}.v{SemVer number} (It’s also allowed to use partial SemVer numbers).
Using it this way, EHRbase also allows updating the templates, even if compositions use them:

  • template_test_name.v1 → template_test_name.v2
  • template_test_name.v1 → template_test_name.v1.0.1
  • template_test_name.v.1.0.0 → template_test_name.v.1.0.1

Note: All template versions would be available in the template list as standalone templates.

Following this template naming convention would also allow compositions to be updated to newer template versions.

For example, a composition based on template_test_name.v2.0.0 can be updated to use template_test_name.v2.0.1, but cannot be updated to a previous template version template_test_name.v1.0.0. Since composition data is immutable, every template change in a composition has to be done via a new update to the composition, so a new version is created.

8

Just so I get this right, the migration strategy for someone using this feature is:

  1. create template_test_name.v2.0.0
  2. create composition A that conforms to template_test_name.v2.0.0
  3. update template_test_name.v2.0.0 to template_test_name.v2.0.
  4. The CDR automatically updates composition A to now conform to template_test_name.v2.0.1, and creates a new version of the composition?

Does the CDR automatically do this, or is the user expected to update the compositions themselves via the API?

9

To add to the conversation in general, I think there are 2 complementary actions we need to think about whenever we’re updating a template.

  1. Update the template definition
  2. Move the composition that were of the previous version to support the new version of the template

I really don’t see enough people talking about how to do (2). We’ve had production data painstakingly moved to newer composition versions overnight as an ETL job before the practice starts the next morning (not fun), so the application can just work with the new version of the template.

I think doing these sort of migrations safely will be an integral part of a long-term CDR, and maybe even the template definition itself. Why?

Let’s compare it to simple SQL (pseudo code):

Someone creates a simple template with field1, field2:

create table template_1 (field1 text , field2 text)

Let’s say they want to add a new field3, which is mandatory. You need to now define the default value that this field will take in the migration definition so that the previous instances can be back-filled.

alter table template_1 add field3 text default ''

Similarly a rename of field2 to field3 explicitly also moves the data that’s stored in field2 to field3:

alter table template_1 rename field2 to field3

Almost no SQL migration just writes another create table with a new definition and calls it a day. The work of moving the old data to the new format is part of the new SQL definition.

At least my opinion here is - it be the job of the template author to safely define how the fields from an older version of the template will move to the newer version of the template - because they probably know best how what kind of data was stored and how to safely move it to a newer version.

I also think this is not really out of scope to define these migrations as part of the OPT definition itself (or maybe another complementary format) that CDRs need to accept when a new change to a template is made.

And before someone says - the composition data is for life, and you shouldn’t move it to another format, and just keep every version in the CDR and just query it using AQL, try telling my devs who now have to maintain AQL like this in the application layer (pseudo code again):

select t1/path1 as pulse_rate_1, t2/path2 pulse_rate_2 ... from template_1 t1, template_2 t2

I’ve had real instances where someone used an older version of an archetype and forgot to include it in the AQL, and the clinician was pissed that it didn’t show up. This approach is not clinically safe. It’s just really hard to keep an inventory of all the different versions of archetypes we’ve used, and all the versions of templates you gotta think about when you query this data.

10

There’s no automated action by the CDR (EHRbase).

  1. Composition A.v1 is based on the template template_test_name.v1.0.0
  2. There is an update to the template, and the new template version is uploaded to EHRbase template_test_name.v1.0.1
  3. For Composition A to conform to the latest template version, a Composition update needs to be performed where the template_id is set to point to the new version, so you end up with Composition A.v2 based on template_test_name.v1.0.1
    1. At the same time, a Composition B is not forced to update to the latest version. It’s up to the users/clients to decide how the changes to the templates need to be reflected in the data.

Here’s a representation.

image
image429×550 44.9 KB

11

For non-breaking OPT changes, data doesn’t need to be migrated. It works with the new minor version of the same OPT.

@Sidharth_Ramesh Your #3 says: “update template_test_name.v2.0.0 to template_test_name.v2.0.” It is missing a minor version number at the end. I guess it should be “v2.0.1”.

I believe your example of adding a new mandatory field cannot occur during a minor OPT version change (it would cause all the existing compositions to fail the validation with template_test_name.v2.0.1).

@vidi42 Your #3 says: “…Composition A.v2 based on template_test_name.v1.0.1”. Is this correct? I guess it should be “…Composition A.v2 based on template_test_name.v2.0.1”?

I believe the CDR should handle Composition A.v2.0.0 even without updating its template_id to v2.0.1. Composition A should be valid for any v2.n.m OPT version available in the CDR.

Adding a new mandatory field requires a new major OPT version. This does cause the issues you described regarding AQL. Adding a new OPT version to all AQL queries is one way to handle this (it doesn’t require data migration) and is how Alex described handling Composition B.

Another approach is to migrate all compositions to the new major version OPT. This approach seems impractical due to the volume of data requiring migration (and the OPT authors not providing a migration plan).

Neither scenario is ideal for the users. In practice only the AQL scenario is available to the clients. The consequences of failing to update all AQL queries are real. I’m not a fan of putting the full burden on the clients. I also believe it will be bad for the openEHR image.

One way I’m exploring to mitigate these issues is storing archetype data using inheritance. For example tables for blood_pressure.v1 and blood_pressure.v2 inherit from the blood_pressure table. This allows using the version-less archetype name in queries (the database includes the data from the v1 and v2 tables). When a new major blood_pressure.v3 archetype is introduced, most queries will continue to work and will include the data for the new version. The good thing is that the queries will fail loudly instead of continuing to work while silently ignoring the v3 data.

The above works only if the new major version didn’t “break” the structures used in the queries. In other cases, the CDR could try to handle it by introducing alias fields and other technical solutions to the version-less archetype table. Some cases will still require changing the queries or migrating data. I would just like to investigate how much the CDR can help with new major archetype versions at a “technical” level.

@Sidharth_Ramesh I believe you encountered this problem because you mapped RM data directly to SQL tables. PostgreSQL is just not up to handling openEHR data with that approach. This is why I picked ArcadeDB which gives us more options when transforming RM to native database “objects”. Inheritance, alias fields, and strictly typed JSON fields are a great help.

12

Yes indeed - meant 2.0.1

The problem I was talking about was with deployments using EHRbase and just plain openEHR REST APIs. Don’t want to bring in my work on openEHR on SQL / Postgres into this conversation.

We really have to pick between these 2 evils:

  1. migrate all compositions to the new major version OPT. This approach seems impractical due to the volume of data requiring migration .
  2. Adding a new OPT version and archetype versions to all AQL queries. This approach seems impractical because application and query complexity will keep growing. It’s also clinically unsafe if not done well.

Now I would really like there to be a magical 3rd option where openEHR “semantically” figures out the meaning of an AQL and propagates it down to older versions of the archetype / template etc. But it just opens up another surface area for bugs.

Maybe some smart people can really sit down together and figure it out? But then the semantics of AQL itself might change. Instead of querying for :

select systolic from observation o [blood_pressure.v2]

maybe we can now do

select systolic from observation o [blood_pressure.*]

But this isn’t possible today because the paths between these different archetype versions aren’t the same.

Maybe in the future, template IDs can be queries semantically with a wildcard on the version. But then someone really need to make sure these semantically meaningful “paths” remain consistent.

13

I meant template_test_name.v1.0.1, because that’s the new version of the template. Please check again, there’s no mention in my coment of template_test_name.v2.0.1. These versions are anyway just there to represent change, not to be taken as hard constraints or requirements.

I believe the CDR should handle Composition A.v2.0.0 even without updating its template_id to v2.0.1. Composition A should be valid for any v2.n.m OPT version available in the CDR.

openEHR doesn’t specify template versioning (apart from some support with 1.1.0 REST API and some latest changes to the template if I remember correctly) or what constitutes a breaking change in a template. So each template is, in the end, a standalone entity. The versioning mechanism is something added on top that is not strictly enforced. Users of EHRbase can decide if they want to benefit from it or not by following the SemVer semantics.

As per my representation, Compositions based on older template versions can still function alongside Compositions from the latest version. Any template change, Composition update, and migration of data should be a conscious decision of the users of the system and handled based on their use cases.

14

It says “Composition A.v2 based on template_test_name.v1.0.1”. It says v2 is based on v1. This doesn’t look correct to me.

@vidi42 You did use v2 here but this is not important. I had to scroll up and down to check which versions Sidharth and you used in your examples :wink:

I believe we all know what we are discussing and can focus on handling new major versions of OPTs.

I believe another misunderstanding is caused by ADL14 which doesn’t support semantic versioning of templates. I’m used to ADL2 which uses full versions in instances.

15

Thaks Borut,

I agree with your comments that minor and patch changes do not need any original compositions to be updated. The logic should be identical to that applied for archetypes - if a path that was valid an iny existing compositions changes or is deleted, that is a major version change and requires specific handling.

I agree it id a challenge for impleneters but is not unique to openEHR - we see the same challenges in FHIR and even in SNOMED when internal relationships need to be updated.

This was also part of @Sam disacussion at EghrCon25 on how to minmise the burden, particualrly as you say, that in many cases, the breaking change is not operationally relevent as the paths being altered were never in the original template.

I’ll come back on this (maybe we should strip this out into a separate topic) as I think it is an important area. I have had an idea about aliasing archetypeIds (part of the ADL2 transition challenge), which may actually help here too.

Could you maybe copy that part of your response into a new topic and I’ll jump in there?

@vidi42 - this is really good news and very much in line with the new SEC guidance/spec on using the HRID (ADL2) pattern to identify ADL1.4 templates - basically what Ehrbase is now doing but with the addition of a namespace. All of the CDR vendors and tooling vendors are signed up to support this change which is just going through final SEC review, so hopefully tooling will make this fairly use to setup before long, though it remains optional

e.g. com.medblocks::template_test_name.v2.0.1

16

There is no magic happening on its own :wink:

@Sidharth_Ramesh I believe allowing version-less AQL queries cannot solve the problem. It has to be at a lower level.

This is how clients might interpret it: “Our CDR doesn’t have features for handling our use case. We cannot handle the migration ourselves and are dependant on the CDR vendor’s willingness to do this for us. We are at their mercy to receive a reasonable quote for this project.”

I’m sensitive about this because I went through this with vendors implementing customized Microsoft’s ERP systems that customers were never able to upgrade. They expected to be safe since they picked Microsoft, but they didn’t expect vendors with self-serving business practices. This disaster affected business data. I can’t imagine the same scenario with health data.

I believe openEHR should offer migration paths for users; however, it cannot because the migration depends on the CDR implementation. I’ll start hoping for some magic :wink:

17

I think we can be fairly safe in applying the same rules to templates as we do to archetypes, particularly as in ADL2 templates re technically specialised archetypes. But is hard to be absolutely didactic about what constitutes a change especially patch vs minor revision. And I think these are decisions that have ot be made/asserted at the modellingf layer not in CDRs

This will be an opportunity to review the various approaches in AD and CKM (or others) and perhaps develop a consistent algorithm, even if that does sometimes have to be manually overwritten

18

I agree with what you’re stating in the rest of this message. But this statement I would like to understand better:

Why can’t CDR’s ‘author’ OPT versions? What do you mean with authoring and CDR?
e.g. the Nedap CDR generates OPT’s from templates authored by a modeller without semantic changes just flattening from adl2 template to a flat OPT in a predictable way. This is fine right?
It also generates an OPT from just an observation archetype, assuming a default COMPOSITION.report-result, assuming the observation is a standard ‘clinimetrics’ score/scale. This is just a shortcut for the modeller and fine as well.
Sometimes updates to the OPT can be ‘programmatic’ in a feature upgrade that changes the OPT, in a migration scenario, coded by a developer. This to me, can be fine as well, as long as the one responsible for modelling is in control of those changes.
Or do you really have different views @chunlan.ma ?

19

I understand “CDR shouldn’t author OPT versions” to mean that only modelers should change the version part of the OPT (e.g., .v2.0.1), and that CDR should not change it when OPTs are uploaded to the CDR.

I guess this is clear to everybody who replied but I just want to say it explicitly: a template can only be overwritten with a newer SemVer (e.g. template.v1.0.1 can overwrite template.v1.0.0). However CDR should reject uploading template.v1.0.0 if there is template.v1.0.0 already in the CDR.

This means CDR stores all the uploaded versions for an OPT, but it doesn’t “version” them.

20

Yes, that is the default position in ADL2 and also if people adopt the HRID pattern for templateIds.