Separating Models from Implementation

erik.sundvall · 26 November 2021 07:29

When creating a generator that parses an operational template definition (in OPT, webtemplate or other format) it would likely not be too hard to create both JSON schema and OpenAPI outputs in the process (they have a lot of similarities). Some tools like Altova Mapforce (and perhaps Mirth?) support Json schema and many code generators support openAPI.

A quick dirty hack POC version of schema+OpenAPI generator might be drummed up basing code on @bna’s web template → Asciidoc script that @ian.mcnicoll has continued on at https://github.com/freshehr/wt2doc (mentioned in post 20 above) and replace the DocBuilder.ts class etc. A better more properly model based converter may could likely be done fairly quickly by @borut.jures and his nice builder code or by something from @pieterbos & Co at Nedap or somebody else.

There could of course be many improved JSON structures to target, but to start with the simplified “structured” JSON that Better, EHRbase and others are using would be good to hav for integration tools that may want to loop over a structure. It’s called structSDT in Simplified Data Template (SDT) specification and COMPOSITION data instances can be both imported and exported in platforms like Ehrscape, see images below of the non-standardized way:

If using the standardised openEHR REST API I hope the same could be accomplished using the string application/openehr.wt.structured+json in the http-header Content-Type in the requests as described in the Alternative data formats section of the REST spec overview, when POSTing and GETting COMPOSITIONS. How many openEHR implementations support this already - ping @birger.haarbrandt @bna @pieterbos @borut.fabjan @stoffe @Seref and others?

damoca · 26 November 2021 07:43

Streamlined?

Existing technology is enough for doing this. GraphQL is a kind of query language for a data exchange API. If we are able to define a streamlined data model that includes every bit of information of the RM, it would be a responsibility of the data consumer to ask for the information or data points they want. I think it is the most flexible approach to deal with the RM extensive metadata plus the variability of archetypes and templates.

erik.sundvall · 26 November 2021 09:01

I agree that GraphQL would be interesting, but to live up to developer’s mental models of it, they (at least the ones here at Karolinska) would likely want to be able to think that there is an active data store behind the scenes where you can pull up a composition and interact with it read+write on a specific field, or on a subtree in chunks, via GraphQL using “simplified” template-specific names + whatever other RM things that are unconstrained/unmentioned by the template. To do that I think a contribution builder approach would be good, where either an existing or new blank COMPOSITONs could be instantiated, read and modified and then, when finished, written back. Can we continue GraqphQL-discussion in that thread?

ToStupidForOpenEHR · 26 November 2021 11:38

Would that not require the consumer to have some detailed knowledge about how the RM works ?

Which is one of the things I am trying to avoid…just want a concrete (flattened) representation like the Mindmap or Printable version from the ckm

yampeku · 26 November 2021 11:42

You could have predefined queries for specific use cases or roles (e.g. what data is useful for research purposes), as @siljelb, @heather.leslie and @ian.mcnicoll have commented in the past

joostholslag · 26 November 2021 11:45

I think we all share that the export would need to be usable without detailed knowledge of the rm. But anyways there’s the question of how to handle mandatory and optional datapoints. The mandatory datapoint in archetypes are used sparsely and should also be hard requirements for the exported model (e.g. a diagnosis without the mandatory name makes no sense and shouldn’t be considered a diagnosis). On the other hand archetypes and the inherited rm attributes can be overwhelming. Could be going to a hundred optional data ‘attributes’ that would make a developer overwhelmed I assume.
Templates and/or Graphql could help handle that problem.

ian.mcnicoll · 26 November 2021 12:07

This is the big challenge and applies to FHIR and other info models just as for openEHR. You can’t ignore that at some level there is going to have to be some common underlying infrastructure, and enough flexibility on the data models to allow them to be ‘finished’ differently in differing contexts, including the impact of whether you are using SNOMED CT.

So the basic models (archetypes/ resources) simply cannot be locked down at international level. they need to be adaptable and that inevitably means that ‘someone’ has to take the responsibility of applying the maximum degree of ‘interop/coherence’ at a more local level., and hiding the more obscure parts of the models, or indeed the sub-structure where they do not apply to a specific local context. And those decisions are what clinician informaticians need to do

This is where the flattening process can be very helpful, it allows people like me to use tooling to generate simpler structures that both hide parts of the archetype that need not be visible to the dev, and to make those important parts of the substructure that are not in the archetype, visible to the devs. We can probably come up with a best-guess at what these might be for ‘most scenarios’ and would give Gary what he wants . To some extent that is exactly what the Ehrscape composition example API call does - it exposes a few RM end-points that are either mandatory or commonly required e.g. participations, subject.

So yes, I agree to get maximum benefit this needs to work at template-level and fold in the three layers - if you just want a ‘raw archetype’ , just create a very simple template with a basic composition, and an unconstrained e.g Blood pressure archetype - export the Web template and use that to generate what ever downstream artefact you want, ignoring the top-level Composition.

borut.jures · 26 November 2021 12:57

Thank you for the kind words about my generator @erik.sundvall . I’ve spent the past few hours writing a generator of OpenAPI schemas.

Here is a separate topic with a download link and source code:

joostholslag · 26 November 2021 13:04

I’m not in favour of obfuscating the archetype/template distinction with exports. I think it doesn’t make sense to offer a very simple template with a basic composition. I think it does however make sense to offer some common usecase e.g. result of a BP measurement as a template (bp observation, result-report composition). And export that as OpenAPI (or json-schema, XML, or webtemplate). But the implementer has to be aware that his implementation has to match the context (actual measured bp of the subject of the ‘ehr’, not e.g. a target bp) of the template, or stuff will go horribly wrong for that patient. It is fine to not constrain out many of the elements in the bp observation archetype, as we’d normally do, and leave that up to the implementer. But mandatory elements should at least stay mandatory (shouldn’t be a problem, BP archetype has none, report-result composition has only very straightforward stuff: e.g. start-time: who would want to record a BP without recording the time of the measurement, that has no value).
This would even be very valuable to many openEHR implementers.

We all don’t think it makes much sense to export only archetypes. Since the implementer will have to supply the context information and this will bring back all the complexity of openEHR. And will realistically never be interoperable with native openEHR. But I also think it’s not up to us to keep people from trying. Maybe they come up with something simpler, better, cheaper, faster than openEHR and I’d be all for that. So we should not block export of archetypes. But realistic, it is not. So I wouldn’t enjoy spending time on helping people, who don’t understand the complexity to do this. And I wouldn’t want to be responsible for contributing to another ehealth related clinical disaster.

damoca · 26 November 2021 13:05

Not exactly, what I said assumes that we have already solved how to generate that streamlined format (a data structure which directly uses readable data element names extracted from the archetype/template). The only documentation needed to know what it contains would be approximately the one you find in the CKM:

ian.mcnicoll · 26 November 2021 13:08

Yes - Heather, Sebastian and myself worked to create that list of 'clinically-relevant parts of the RM and would be a good candidate ‘defaults’ to be folded into the flattened model, although like Silje I would like to be able to override those defaults for a particular use-cases. We do actually have the tags to be able to do that in AOM2/ADL2 - just nor implemented in AD / .opt yet.

joostholslag · 26 November 2021 17:26

I’m curious for that list.

ian.mcnicoll · 26 November 2021 17:56

Each CKM archetype and Element has a Reference model tab with a short list of ‘clinical’ RM attributes - a bit arbitrary but probably close enough.

@sebastian.garde - are you able to share the underlying XML file?

thomas.beale · 27 November 2021 18:12

There are even markers in the RM BMM to indicate which RM attributes are interesting and which really are not - ADL Workbench filters using these and can generate any variant of RM flattening onto a template or archetype. It’s easy to put more classifiers like that in BMMs generally.

Example here.

erik.sundvall · 28 November 2021 22:08

They are all template-specific “streamlined” things, I belive TD… = Template Data…

TDS …Schema (XML schema)
TDD …Document (xml instances adhering to a TDS)
TDO …Object(s) (Class definitions, used for simplifying programming applications & integrations)

The template specific “streamlined” OpenAPI descriptions discussed in this thread could be used to produce the equivalent of TDO in any programming language (I believe TDO from the back then was usually in some .NET format)

@heath.frankel is likely the main guru on the topic and was involved in creating TDS/TDD/TDO formats many years ago at Ocean Informatics (now called https://oceanhealthsystems.com/). I believe many of the reasons for their creation were basically the same as the ones @ToStupidForOpenEHR brought up and that started this thread - to make it easier for developers and others to, for a specific use case, integrate/interact with openEHR systems - using familiar tools and without a learing too much about openEHR or openEHR specific tools.

When searching for TDS in the old openEHR mailing list archives you’ll find a lot of the same thoughts as in the current discussion, e.g. in 2008 and 2013. I believe the main problem regarding adoption was that the TDS/TDD approach was never published as an official specification back then - it was used by Ocean and some of their customers (including Better/Marand for example).

In 2016-2017 things had changed and there was a willingness from Ocean to share/publish the approach and create a specification, work was started, see the wiki page https://openehr.atlassian.net/wiki/spaces/spec/pages/30408770/Template+Data+Schema+TDS+Specification+and+associated+Template+Data+Document+TDD but never finished because of peoples’ busy schedules. Heath (who was very busy then) sent me some files that I (also busy) never got around to analyze and publish on that wiki page as a basis for further standardisation work. Perhaps it is time to pick that up again and let anybody interested help documenting the approach? @heath.frankel could you attach any files/descriptions for curious people to look at - on that wikipage or in this thread?

Others may have more information and links about TDS/TDD/TDO, please share. @Seref may know of current use at Ocean.

ToStupidForOpenEHR · 29 November 2021 09:58

Thanks @erik.sundvall

TDS/TDD does look fairly close to what I am suggesting.

erik.sundvall · 2 December 2021 10:02

In order to not kidnap the thread about @NeoEHR’s cool generators I’ll clarify some thoughts here instead, where I believe the core of this discussion fits better.

@borut.jures and @thomas.beale I am not sure I explained my point well enough in OpenAPI schemas generated from BMM files - #9 by erik.sundvall when refering to @heath.frankel’s comment from 2012 on the wiki page https://openehr.atlassian.net/wiki/spaces/spec/pages/4915205/Ocean+EhrGate+-+Composition+Builder+programming+example

Yes developers/integrators need to learn a little bit about openEHR basics (but not much) if they e.g. have a template-specific use-case that is one of their first tasks to interact with an openEHR system.

However it would be great if they did NOT have to switch tooling or start using openEHR-specific path-based ways of interacting with components like the one suggested in the 2012 wiki page or the more elegant/compact toJson() method described by @borut.jures in OpenAPI schemas generated from BMM files - #10 by NeoEHR

By tools/methods they are already used to, I mean…

for integrators/informaticians creating and maintaining integrations: low code tools like Altova MapForce or Mirth as described in Graphical data mapping tools supporting openEHR? (these tools work best if supplied with a template specific schema with user friendly node names derived from the template labels - TDS is one such approach, a similar way to do JSON schema for the “structured” (JSON tree) simplified format described as structSDT in the end of section 3.2 of a spec would also be a great option for such tools
for software developers: a template-specific litte library of classes with names derived from the template labels. TDO is/was such an approach, template specific openAPI definitions would do the same job in a more modern and programming language-neutral approach. Then they can just generate and import the library/classes and keep using their favourite editor/IDE and get syntax higlighting, auto completion, validation etc based on the template (Instead of string based path/key approaches where errors often are only found at runtime).

So it’s not that we would try to “hide” everything in openEHR, it’s about making the integrator and developer experience more efficient and less error prone.

Edit: @Seref pointed out in a recent SEC-meeting that the simplifications in TDOs etc stopped somewhere around the openEHR data-type lower levels, and there reused a generic part for different use-cases/templates. Perhaps @seref or somebody else that is used to working with this could explain further in a few sentences?

ToStupidForOpenEHR · 2 December 2021 10:43

You also have to consider the case that HealthCare providers that I work with e.g. NHS Scotland, NSS, Boards, Trusts across England, NHSx, NHS Wales, eHealth Ireland do not have particularly big or strong development teams, and almost universally work with .NET. So trying to uplift OpenEHR in a primarily JVM based world is a real challenge.

It’s also very rare for a developer to be allowed to spend time learning a technology without this being part of the strategic roadmap for the provider.

This goes right back to my initial point, I want OpenEHR and in particular the models to be widely adopted, but HealthCare providers (boards, trusts etc) are put off by the complexity of OpenEHR, so are not including OpenEHR in their strategic roadmap.

I’ve taken a fair amount of abuse trying to get this point across. It’s nothing about me learning OpenEHR that would not change a thing. It’s about getting someone like NHSx to adopt it strategically across NHS England.

Myself and colleagues are pushing OpenEHR on the policy front not only in the UK, but across EMEA and in Australia and NZ.

We are trying our best to get OpenEHR adopted.

Joost and I had a chat, and we think that OpenAPI is a good way to go.

If you want to help then please let us know.

Gary

Seref · 2 December 2021 13:18

I’ll respond carefully and in case I fail to reflect the history/motivation accurately, @heath.frankel is welcome to correct me.

Both the TDS/TDD and TDO approaches were Heath’s babies. He’s the one who came up with the idea, and the aims were at some level overlapping with what’s being discussed here: make it easy for non-openEHR systems/actors to talk to openEHR, make it easy to work with templates for a developer.

In case of TDS/TDD, Ocean’s template designer produces an XSD from a template, called TDS, and if you can produce an XML file (called TDD) that is valid according to that XSD (schema). This approach dealt with the interoperability requirements of the time neatly. Back then, XML was the dominant ITS for exchange, and an XSD and an XML file did not take much explaining. Sure, XSD could not express all the constraints expressed by a template, but it was a good balance between stopping openEHR leaking to non-openEHR world and validation at source. When TDD(XML) arrives at Ocean land, it is transformed to a canonical Composition XML, then deserialised to RM and validated. Ocean template designer is still freely available after all these years, so you can play with it. The transformation to canonical XML is done via XSLT but I’m not sure if we made that available publicly.

The slight problem with TDSs is, they’re fully self contained. As in, each XSD creates its own little type universe, and with xml namespacing etc, two DV_TEXTs in two XML files end up having different XML types from two different schemas. Not a real problem because a TDS is meant to be a messaging artefact, but in the context of conversation going on here, it is something to keep in mind. Actually, the openEHR SDK from EhrBase had the same behaviour (or I think it did): it generates a set of Java artefacts from models and same RM types end up with duplicate, yet technically different representations in generated code, because they’re based on different opts. Again, corrections are welcome.

A TDO is actual C# source code, generated from templates, and it allows a developer to populate data using the names/concepts as they’re used in the template. This makes a big difference in terms of productivity, but Heath was concerned about various software engineering aspects of using TDOs so he was not entirely enthusiastic about using them, and I disagreed If my memory is correct, when they started using openEHR based on the Ocean platform, Marand (Better now) even produced Java TDOs and used them for a while. Happy to be corrected on this if my memory is wrong.

The tricky bit with TDOs was that they required code to go from the TDO type to a canonical RM composition, which is what’s done with XSLT in TDS scenario, and that functionality was not convenient for us to distribute as a library, though at some point around 2014 or so I remember doing it for a partner in an Australian project.

As of today, we have TDOs in our production code, I’m pretty sure of that, and one or more of our integration endpoints may be using TDSs; this I’d need to ask Chunlan Ma, who’s the queen of openEHR implementation

So as you can see, the idea of isolating (which I prefer to separating as a term) openEHR from the non-specialised developers is something we’ve been experimenting with and doing in different ways for some time.

I personally think it is a very good idea, and the eco-system of today provides a lot more options than the time Ocean did TDS/TDD: that’s the thing with arriving too early, there’s nothing around for you to leverage…

This thread has enough disagreements without me pouring gasoline over it, so I’ll keep my specific opinions to myself, but as someone who does this for a living, I see a worrying gap between off the shelf developers and their practices and what we’re doing. Yes, we’re taking steps, but there’s a long way to go and I see lots of good ideas being exchanged, so hopefully we’ll do better come this time next year.

thomas.beale · 2 December 2021 17:01

Right - but one has to ask - do they even realise the problem they need to solve is fundamentally complex? If they don’t, then there’s nothing to say, they will fail, guaranteed. But if they do, really, they need to hire proper engineers and developers to do some core work, e.g. deploy the platform, learn the platform API, and how to do a few things with templates. If they don’t want to even do this, and the current employees simply want an easy ride with everything so ‘simple’ they don’t have to do any work, well then I am afraid there is no path to success. They will continue to fail as they have been for decades, apart from a couple of centres of excellence like Marsden (Jo Milan).

I hate to re-iterate the question of complexity, but it will not go away by complaining about it. If NHS trusts cannot realise what kind of problem they are solving, they have no route to success. None.

Now, there is no doubt that we have not quite arrived at the optimal point of application development today - the things that @erik.sundvall and others are talking about (and web templates, created by Better, TDS etc by Ocean in the past) are needed, in an updated form. (As you can see, there is furious activity in this area).

BUT. If the NHS wants to be serious, it needs to invest in this phase of technology development. Unlike the millions they waste on closed source EMR systems that don’t work, every penny they spend on openEHR is public IP and costs nothing once developed (other than the cost of maintenance obviously). The central question is: are they serious?

It’s not a technology question. It’s an organisational question.