Best practice for updating templates

Implementation Platform
1

Hi all,

I’m new to openEHR and have been playing around with ehrbase. I am wondering what is the best practice for updating a template that has been uploaded to an openEHR server.

I have noticed in the REST spec that there is no DELETE or PUT/PATCH endpoint for templates, and in my experimenting with ehrbase it prohibits me from uploading a template with the same template_id. My impression is therefore that templates can not be deleted or updated and the template_id must be unique (makes sense).

From looking at other templates and archetypes I named my test template ID acme.test-consult.v0(i.e. including the major version but omitting the minor and patch) however in order to make a change it seems I need to change the template_id. Should I put the full semver-style version as the template_id? E.g. acme.test-consult.v0.0.1? I haven’t seen the full version name included in any other example, but it’s the only way I can seem to upload a new version of an existing template. I noticed that a lot of archetypes have a revision field that contains the full semver-style version, however ehrbase at least does not seem to look at this.

Thanks!

1 Like
2

Hi @jessarcher,

we are currently working on the Admin API in EHRbase that will allow you to delete and update Templates. There is also a spring configuration that allows to simply overwrite templates with the same ID. You can find this one in the application.yml

image

or you provide it as a parameter if you are using Docker:

… -e DB_USER=ehrbase -e DB_PASS=ehrbase -e SYSTEM_ALLOW_TEMPLATE_OVERWRITE=true …

Using semver-style is advised. When overwriting templates the challenge is to ensure it is backwards compatible and EHRbase unfortunately does not have these checks in place at the moment (I think the Clinical Knowledge Manager has some means to distingush between minor and breaking changes).

Hope this helps!

1 Like
3

Hi Jess,

Welcome - good questions and I agree with Birger’s response. We figured out how to apply the server config to a Docker install so let me know if you need any advice on that.

Quite clearly, once we are in a production phase, we really need to keep tight control of versioning, so not allowing updates via the REST API makes sense but this is pretty burdensome when in the development stages, particularly if one is rapidly iterating the UI and making changes to the template in response, or (as in the CVID scenario) where the template is changing fairly regularly in response to fast-moving requirements changes.

The semver numbering approach has served us well for archetypes but for various reasons has not yet been applied to templates. However, I think the general approach should be the same (certainly my policy) - add the Vn major version number to the templateId.

You can add minor and patch numbers to the templateID but you then get into problems of making it difficult to apply non-breaking changes. I think the principle that we applied for archetypes is best. Use a separate semver number to manage more subtle changes.

When we finally get moved to ADL2 for templates there is a formal place for semver , but we don’t have this now in the .oet/.opt files. I think it is time for us to agree on adding that as a ‘standard’ annotation
which would allow the semver to be carried until such time as we make the jump to ADL2.

and welcome to the community :slight_smile:

2 Likes
4

One general request for feedback: we are currently implementing the delete template function in a way that we forbid it as long as templates are still being referenced by compositions.

Furthermore, we are thinking about a mechanism that provides versioning on templates similar to versioned objects in openEHR. This means that updating means not overwriting but creating an active version and still keeping track on the previous versions. This way, it is possible to re-create the state of the database at any point in time regarding the template used. Good idea, or do you think this is not too useful in reality?

1 Like
5

Thanks for the great responses Birger and Ian :slight_smile:

I imagine Ian will have far more insightful feedback, but I can provide my early perspective.

With regards to deleting - the rule you’ve outlined sounds great, although I don’t currently have too much of a use case for deleting templates, especially with the overwriting functionality that you’ve previously mentioned.

I am very interested in template versioning and having the ability to upload new versions of templates. Both with backwards compatible changes, but also with breaking changes. Similar to the archetypes on CKM.

I have been involved with creating clinical forms and am currently investigating how openEHR might fit for us. I’ve witnessed a lot of change requests to our forms (mostly during UAT, but sometimes after launch as well) and I am trying to work out how best to handle that with openEHR when the form change requires a change to the underlying template.

I’m still in the proof-of-concept phase though. So I’m not sure yet how problematic it would be to just include the full semver in the template_id and effectively create a new template for every change. I’d just need to make sure that all new compositions use the new ID.

From an AQL point of view I think I should still be able to run queries like “give me all the blood pressure readings for an EHR” regardless of what template was used when the data was recorded.

1 Like
6

What Birger has proposed is a great approach. That way the older versions will not be actively available, but still will not break any data already in the CDR.

  1. Delete only if no reference in any composition
  2. Version update for those in service already so that old version is not in service anymore, but available in the system.
    regards
2 Likes
7

Hi Jess, Good discussions,

From experience and would countenance against adding the full semver to the templateId, especially in the development phase. Inevitably there is a lot of to and froing between the app developer and the template developer, particularly if you are using form builders. It gets even more messy if you are uploading to a tool like CKM which works on the templateId as a key.

My current approach in development is…

  1. Just put the major version in the templateID name.
  2. Add the semver to the other_details metadata sem_ver =0.n.n asa stop-gap until ADL@ allows us to record that ‘correctly’.

I have started a conversation with @borut.fabjan of Better and @sebastian.garde of CKM about how we can get tooling to help us here.

Semver has worked very well for us at archetype level, since it is really helpful to understand the nature of a change major, minor etc and this will be equally helpful at template level.

One decision we took for archetypes was that for a V0 archetype, the minor and patch numbers really had no defined meaning i.e v0.2.3 the .2.3 could not be relied on as any number of breaking changes may have occurred.

This probably makes sense in the context of CKM but I certainly found it helpful, when doing rapid template iterations as part of covid-19 app development, to be able to flag a potential breaking change to the app developers by incrementing the minor number to signify a breaking change , and incrementing the patch number to signify a minor revision (albeit that the overall published state is still V0). Sebastian and I were discussing that earlier this week - semver itself seems not to care what you do with V0 numbering so we are not breaking any rules there.

Essentially for a .v0 we shift the numbering one to the right, and drop patch.

We definitely have to get smarter tooling here that interrogates the semver number but I’d keep the minor and patch numbers out of the templateID.

The other issue is whether versioning rules apply the same for templates and whether that just reflects a bubble-up of changes in the underlying artefacts. e.g a major version change in a child archetype → major version.

There are complications where a minor change in a child archetype may be constrained out in the parent template, and as such is applying a real-world revision but after playing with a few scenarios we reckoned that it is safer to apply the rule consistently , and allow the tooling to flag that the change applies to an unused node, and let a human being decide if a minor version change should actually be applied, or just apatch. Currently that’s how CKM operates - makes a suggestion but allows human override.

3 Likes
8

Thanks Ian!

I definitely agree it would be nice just to have the major version in the templateID.

How are you currently making changes to the one template ID though? Via the SYSTEM_ALLOW_TEMPLATE_OVERWRITE=true option in ehrbase?

And are you keeping prior versions of the template somehow, or do you find that you don’t need it?

I agree regarding v0 - semver.org says:

  • Major version zero (0.y.z) is for initial development. Anything MAY change at any time. The public API SHOULD NOT be considered stable.
  • Version 1.0.0 defines the public API. The way in which the version number is incremented after this release is dependent on this public API and how it changes.
9

Sometime ago we chose to normalize the OPT 1.4 template_id to be able to better manage OPTs in the EHRServer, including the explicit major version number in the template_id: https://github.com/ppazos/cabolabs-ehrserver/wiki/Template-Management-(from-EHRServer-v1.3)

So when you upload an OPT to EHRServer, openEHR Toolkit or Atomik, the template id is always normalized to concept.lang.version

Also, in EHRServer/Atomik, we have a minor version that is managed internally when you upload many times a template with the same UUID or template_id, which is useful to keep tracking of template development and testing.

I proposed to use such schema many times on the SEC, but it’s not been a priority for the SEC, I guess each implementation does it differently. In ADL 2 there is no problem because template IDs are like archetype IDs. But in OPT 1.4 we don’t have a standard solution, and most implementations are still in OPT 1.4.

2 Likes
10

Good thread, thanks everybody! Some questions:

  1. If we combine the suggestions from @ian.mcnicoll (only major version in identifier) and @pablo (add language code - e.g. sv for Swedish) do we then get identifiers like mytemplate.sv.v2 for language specific template exports and…
  2. … mytemplate.v2 for multilingual template exports?
  3. Have you done anything special regarding file names when storing OPTs, web templates etc in Github (or other public repo/version control system) before uploading to an openEHR CDR? E.g.
    • Do you have a constant filename without version (mytemplate.sv) and only change the internal identifier inside the file or
    • do you create a new filename for each major version (mytemplate.sv.v2) or
    • do you also create a new filename for each minor/patch version (mytemplate.sv.v2.1.3)
    • or something else?
1 Like
11

Consider this was designed only for templates in AOM/TOM v1.4, since those have a single language. For AOM 2 where templates share the same model with archetypes, there is multi language support.

For us the advantage was also to see the files like:

templatex.es.v2.opt
templatex.en.v1.opt

Having the language in the file name helped with file management too.

2 Likes
12

We tried…

…and it gets pretty messy to work with (a directory content mayhem plus a lot of renamings to do) so we’ll likely go for something using git tags instead and perhaps combined a branch for each family of derived files maintained together.

Has anybody tried something like that for derived files (Like OPT) in Git?

13

In any case the version management would be done via a system (git or whatever) and the file name should be just an output of that management. Directory listing for branch versioned files via a system could have the logic to sort the files in a smart way were in a file system would be just listed by their alphanumeric order.

Normal

…v2.1.1
…v2.1.11
…v2.1.2

Smarter?

…v2.1.1
…v2.1.2
…v2.1.11

In the template management system the physical files might not exist at all, it depends on the design. I use a mix of database metadata + files on my template management systems in EHRServer and the openEHR Toolkit. The file names are UUIDs, and saving a template file, we only have the major version in the file name. The minor and patch are saved only in the system that manages the templates. But all this was a personal design decision, and is about template management, and I needed a more formal way to express template IDs than just open narrative text.

14

Assuming you’ve solved, or at least come up with an approach for how to version your templates, how are people dealing with migrating the compositions?

Say I have version 1 of my template that I’ve used to capture a composition for a patient.

We add a new field to the template and publish version 2.

This is a non-destructive update to the template and the composition from version 1 would be compatible with the new template (as all the same fields are there, there’s just one new field). How have you gone about migrating/moving the composition to version 2 of the template?

This quickly becomes a larger issue/topic when we start talking about more consequential changes, like fields being moved, renamed, removed, going from optional to required etc.

We’re beginning to start exploring this topic and would love any insight from the community on this aspect of versioning.

1 Like
15

It depends on wether there’s a need for old data to be migrated to the new template version (e.g. for persistent compositions).
@Jelte could share some details as to how we’re doing this at nedap.

1 Like
16

A good question @joostholslag.

Perhaps some naivety on my side here, but let’s say I have a “family history” template that is collecting persistent compositions.
Over time, standards and/or opinions on what should be collected might change and we might add some new fields to our template and remove some.

We “update” our template by publishing a new version, but how do we go about bringing the existing compositions that were based on the old template into the new template.
Assuming most of the fields are the same, you wouldn’t want someone to have to re-enter all that information, and similarly, you don’t want them permanently staying on the old version of the template.

If I were to try and simplify it down for our use case, we treat templates as “forms”. They’re filled out when you first create the composition, and when you come to edit it, you’re back on the “form” with all the previous information “hydrated”.
But when we publish a new version of the template, the “form” is slightly different now, how do we go about “rehydrating” the new form with the old data.

Is this a problem you’ve faced in practice?

17

Thanks for the detailed description, very helpful. Yes, that’s been exactly our problem. What we did is run a script to copy all the old stored compositions into the new compostion based on the new version. It worked mostly well but I found it a bit inelegant.
Let me know if you are eager for my colleagues technical details, I could sent him a notification by email. But I think my summary may be all you need:)
I’m curious for other CDRs. @birger.haarbrandt @sebastian.iancu @matijap @bna

1 Like
18

Thanks @joostholslag - that’s very helpful.

We’d initially ruled that out as we deemed “unattended upgrades” a potential for clinical risk - but perhaps we were premature and overcautious.

In either case, it’s good to hear how others in the community are doing; you’ve given my team and I food for thought.

19

These will normally be past versions of the same Family History Composition, which is a Versioned Object (i.e. a versioned ‘persistent Composition’). There’s no requirement that all Versions of a Versioned Object are based on exactly the same template. For persistent Compositions - including things like Medication list, Allergy list and so on - only the most recent version matters - that contains the currently known truth. Previous versions would be very infrequently accessed - usually only as part of some forensic investigation into the state of the data at some previous time.

Unless the template is radically different, the paths of all the old data will match the same paths in the new template. In this case, the old version of the template is computationally conformant to (= minor version behind) the new version of the template. If the new version contains breaking changes, it is in ADL2 considered a different template. To convert data from a template to a new version with breaking changes requires an algorithm to be associated with the new version, that knows how to consume one or more previous major versions and populate the data correctly according to the new version.

These rules can all be computationally enforced in ADL2 because a template is just a special kind of archetype and it is relatively easy (and implemented in Archie) to do a diff and determine if the changes are breaking or not (or the authoring tool will know, and will have created the right kind of identifier for the template).

@pieterbos can explain in more detail if needed on how this all works in the Archie library.

ADL1.4 systems should do their best to simulate the above behaviour.

3 Likes
20

I appreciate that this is an older thread, but has there been any change since? So to summarise: the best practice is to version the template (.opt file), and POST it to EHRBase. And you can set allow-template-overwrite = true if you want to allow the previous template with the same name to be overwritten.

Is there any way to load the template on startup ie. is there a value in application.yml that refers to a list of templates to pre-load?