Simplified JSON data formats for openEHR

[thomas.beale]

The JSON ‘templates’ referred to here are close-to-runtime data instances (i.e. just need the multiple fields expanded and values filled out) of an OPT, but in this ‘compact’ JSON format which programmers seem to prefer, and which is definitely more efficient than standard JSON.

To get an idea of what this kind of JSON looks like, see the Simplified Data Template example (the Lab one) in this draft spec.

That example has programmer-friendly paths, not just compressed data values, and was originally developed by Better (i.e. Marand).

Just to be clear, the wiki page being talked about here is just trying to reach agreement on ‘compact’ JSON form(s) the leaf data items. It’s a work in progress…

[ian.mcnicoll]

In the stringified datatype format for DV_CODED_TEXT, I wonder if we need to distinguish to use cases…

1. Where the rubric/value (text) is intended to be read-only /advisory i.e the original intent of

local::at00034 | Large cuff |

or just

at00034 | Large cuff |

as Better allow in AQL and is very useful in other documentation /visualisation

2. Where the text/value/rubric is intended to be written into the data

e.g the Ocean Temple Designer shortcut for default values

local::at00034:: Large cuff

I’m not clear that we need do both variants - perhaps the context of use makes it clear??

[wouterzanen]

I had the need to look into this because we’d like to transition from the better API to the openEHR API. Currently we can only use the canonical model for compositions and as we work with partners that don’t need to know openEHR this is a barrier.

In my experience if you want to simplify, you will need to get rid of archetype id’s for the simplified model. I like the FLAT model as proposed here (flath , directory path stile JSON), however we should also offer a structured JSON for s simplified model. We need this because most JavaScript Frameworks nowadays can easily access structured JSON paths, which makes it simpler to work with.

I would very much like to help out with developing this idea (although I lack the in-depth technical knowledge). Is there a timeline for developing this further?

Gr,

Wouter

[thomas.beale]

We have created a draft spec here.

Your input - review / suggestions would be most welcome.

[ian.mcnicoll]

The adoption of the Better FLAT format and web template by EhrBase (with some assistance from Better, I understand), gives us a clear path, to make use of this as a de-facto V0 of the draft spec Thomas pointed to.

My current dvice to clients is to use the openEHR REST API for all of the resources other than /composition - where they should use the ehrscape / ethercis endpoints which support the FLAT formats. In due course, I’m sure everyone will move to the formal V1 spec of the Simplified Data Formats but we can anticipate this will take a little time, and may well introduce breaking changes.

Note that EhrBase currently only offers FLAT not STRUCTURED but based on work down by Rob Tweed of Ripple, I am increasingly confident that there is a standard algorithm that will switch between the 2, without reference back to the web template or .opt.

[erik.sundvall]

Can we here continue the thread regarding Simplified/Serial JSON formats and associated REST call headers and content-types/mimetypes) with input from some recent openEHR SEC discussions Feb 22 and March 8, to make it easier to include more people (in addition to the SEC)?

@birger.haarbrandt and I discussed this issue briefly (March 3) and he mentioned that we now have open sourced solutions from both Better and EHRbase that can be used as a reference to point to from specs. There is also a set of conformance tests from Better that they both use. Some related links follow (more details will likely be added later)…

Documentation and/or Source code from EHRbase:

• EHRBase simSDT description 2.5. Step 4: Load Data — EHRbase documentation
• …

Source code from Better:

• GitHub - better-care/web-template: WebTemplate implementation for openEHR CDR
• GitHub - better-care/web-template-tests: WebTemplate unit test suite
• web-template/src/test/kotlin/care/better/platform/web/template/converter at master · better-care/web-template · GitHub
• GitHub - better-care/openehr-rest-tests: OpenEHR REST integration tests

Documents to check for possible change points in, e.g. add references to the open source implementations/tests, update references/examples of formats and REST APIs:

• Serial data formats
• Simplified Data Template
• Sim B
• Wiki page about Simplified Serial Formats
• The REST API specifications, see below
• …

Possible REST spec change points

• General info in API Overview mime types etc. The application/openehr.wt.flat+json as already suggested likely works, also maybe the application/openehr.wt.structured+json should be added. (Any possible later version of the format could be called …wt2… etc.)
• The /composition call in the EHR API needs a way to add the name (ID) of the template, a suggestion is to use a HTTP header named openEHR-TEMPLATE_ID (named after the corresponding class and prefixed as other openEHR-specific headers in the REST API) or alternatively openEHR-LOCATABLE.archetype_details.template_id We might also want to add request/response examples.
• We’ll need to add a way to download “web template” representations of Operational Templates, likely under the definitions API endpoint /definition/template/adl1.4/{template_id} in the REST spec. Would application/openehr.wt+json work? The thing returned would be the same Web template representation format you get from EHRscape API Explorer see image below to navigate to the right spot. EHRbase uses the same format (documentation including pedagogical example) and it is very useful for creating user interface code/tools. You can also download templates in this format from the Archetype Designer GUI
  
• Suggested by @ian.mcnicoll below: IF any of the open source implementations contains an example implementation, then /definition/template/adl1.4/{template_id}/example could be added with behaviour corresponding to /rest/v1/template/{templateId}/example in the EHRscape EHR API (Maybe needs to formally go into in version 1.1.* of the REST spec since it is a new endpoint?)
• If .../example is added, then maybe also a call corresponding to the EHRscape /rest/v1/template/{templateId}/fields should be considered?

(This is a “wiki” post, so others than the original author can edit it.)

[ian.mcnicoll]

The 3 pressing issues IMO are agreeing…

1. How to work with the FLAT compositions via the openEHR REST API -agree additional Content-types, agree how to carry extra parameters like templateID and commitername.

2. Add Get WebTemplate from CDR - as per Erik’s last requirement ? an Accept header?

3. Add Get ‘Example composition’ calls as part of /definition - @pablo has been doing something on this for xml and the Better /template/examples really unlocks the struggle that people have to understand what a composition object should look like.

[sebastian.iancu]

Corrections, additions and other small changes on Content-Type and content negotiation (in API Overview) or extra headers (in EHR Api) can be done easily as part of upcoming Release 1.0.2, as long as that is the actual reality on how the real implementation works (which I believe it is).

Similarly, to add an extra header and examples under Definitions API might be ok. But I just wonder if:

1. the WebTemplate is conceptually, from the perspective of a RESTful resources, the same as an OPT-Template or not ? …otherwise we should add a new definition type (i.e. /definition/webtemplate/v1/{template_id})
2. if it is another format for an OPT, agnostic of the ADL language being used, then we should have /definition/template/webtemplate/{template_id}
3. if is just another representation of an OPT1.4, then perhaps only an extra header on /definition/ad1.4/{template_id} would work, and is doable for Release 1.0.2

Which one from the above you think is more applicable?

About an extra /example as a subresource of template is a good idea, but I would do that as part of Release 1.1.0

[ian.mcnicoll]

It is directly derived from the .opt but re-formulates the content (not based on AOM) to be much easier to parse (from a third-party dev POV) and also adds the shortened path segments that the FLAT/STRUCTURED composition formats use.

@sebastian.iancu

if is just another representation of an OPT1.4, then perhaps only an extra header on /definition/ad1.4/{template_id} would work, and is doable for Release 1.0.2

I think it is close enough to justify just an alternative Content-Type in line with the discussions around adding support for FLAT compositions

Content-Type: application/openehr.wt+json

THis would work nicely with the existing xml .opt : Content-Type: application/openehr.wt+json
or any future JSONified .opt (pure AOM) Content-Type: application/json and be very consistent with the proposed approach to /composition

[erik.sundvall]

There might be one piece missing to make openEHR web application serving easier…
Do we need to recommend (or standardize) a way to store/retrieve versioned additional resources commonly needed for serving UIs, for example form definitions (to be retrieved by form renderer components), images used as UI elements etc?

One way could be to recommend a path and pattern similar to the GET adl2 template adl2 template: /definition/template/adl2/{template_id}/{version_pattern}, so something like /definition/ui/{id-of-the-component}/{version_pattern} (and return latest version when version pattern is omitted).

Another (likely wiser) suggestion would be to not reinvent the wheel and instead recommend using something already standardized for this purpose like WebDAV (or a subset of it) that is already supported by many (web) server and client products (and even naively by many operating systems). If a WebDAV recommendation is considered a good idea we could include that in the 1.0.2 release. If so, should we also recommend a URI mount point for the openEHR-related webdav-repository like /definition/dav/ or not?

P.S.
If comparing to Better’s EhrScape we are talking of storing things like the …/form API call does:

[birger.haarbrandt]

Hi Erik,

the question is if the should be the scope of the openEHR standard. From my perspective, the layer above the platform should be more losely coupled to EHR specification. Would be great to get some opinions on this first before going this route. We need to make sure that besides all the standardization we still have enough room for quick innovation and degrees of freedom.

Birger

[erik.sundvall]

Yes, I am not at all sure we should say/recommend anything about accessing resources/files with content that is not covered by any openEHR standardization. Only asking.

(Looking deeper into for example WebDAV, the way to access a specific older version might not be super-obvious to the average developer.)

[thomas.beale]

I think the general idea of having a standard way to reference UI artefacts associated with semantic entities is worth thinking about. In Task Planning, tasks can refer to ‘form ids’ and/or ‘template ids’. The former is the one an application really wants; for now, it’s just an open string field.

[sebastian.iancu]

Good discussions, but let’s focus now on simplified format here, UI related things can be discussed on a separate thread.

To amend REST specs, I created [SPECITS-57] Update info about simplified format - openEHR JIRA and [SPECITS-58] Add support for /example sub-resource - openEHR JIRA

[wouterzanen]

It would be good to have these simplified formats as my experience with the better formats is very good I am happy to see this moving in that direction.

In the get template call we might consider adding a header for template language so you get the template in a certain language.

It would be perhaps a good idea to work on providing a JSON schema (under /definition) variant based on the webtemplate most information should be available in either the webtemplate or the opt.

[erik.sundvall]

Great suggestions @wouterzanen. We’ll have to discuss with implementers (ping @birger.haarbrandt and @matijap) what goes into first vs next version. We want to avoid delays of a first version.

For language choice, do you think the standard http header Accept-Language would work for requests, with a Content-Language header in the response if requested language(s) was(were) available - and returning a 406 (Not Acceptable) error code if the template was not available in (any of) the requested language(s)?

So a common use case will be that one specific language is requested and you’ll either get a response with it or a 406 error if not available. And if you don’t send any Accept-Language header in your request you get some kind of default or all languages?

Should the above language feature…

• …only be for the calls /definition/template/adl1.4/{template_id} and the /definition/template/adl1.4/{template_id}/example (and /rest/v1/template/{templateId}/fields if added)…
• …or also apply to (some of) the .../composition calls in the EHR API?

@wouterzanen, regarding your last suggestion about JSON schema, I guess you mean a schema defining the rules to validate instances (EHR content) of a particular (web-)template’s flat or structured data instance form? There is a mime-type application/schema+json for JSON schema but in order to separate flat, structured and future other formats without adding extra API calls we could use http Accept headers like application/openehr.wt.flat.schema+json and application/openehr.wt.structured.schema+json in requests to the the definitions API endpoint /definition/template/adl1.4/{template_id}.

Would that cover the needs?

[Sidharth_Ramesh]

I am still confused about which format will be considered the “default” simplified JSON format.
I see in the wiki that the “compact format” is preferred, but I’ve never come across any CDR offering that uses this. I think having a string only representation of the value like “[snomed_ct(3.1)::3415004|cyanosis|]” has it’s advantages, but for most use cases it’s easier to work with Better’s flat format.

Also, is the “Preferred compact format” flattened or structured?

Also, is it possible to use flattened paths, while keeping the value as structured json like so?

{
"path/obtained/by/concatenating/ids": {
  "|code": "238",
  "|value": "other care",
  "|terminology": "openehr"
  }
}

We definitely need to work more on the documentation for these simplified formats. Deciding which one will be the default will help a lot.

[sebastian.iancu]

Trying to clear out things:

• the wiki page you indicated is not a spec, just a SEC workspace we have working on the flat-format subject, so a placeholders for thoughts, etc
• the format specs are in DEVELOPMENT state, and located here: Simplified Data Template (SDT)
• these specs are also referred from latest (upcoming Release 1.0.2) of REST API specs: Overview # alternative-data-formats

Nevertheless, documentation might not be sufficient or not clear enough - therefore we’re happy to receive feedback.

[ian.mcnicoll]

It is a bit confusing, though as @sebastian.iancu points out this is very much work-in-progress.

The initial focus, that I think we have pretty well nailed down is to support the current Ehrscape FLAT and STRUCTURED OUTPUT formats via the openEHR Canonical REST API / composition by adding some headers and Content-Types.

I don’t think there will ever be a preferred format as such, and I’m not sure there needs to be.

The ECIS flat format does introduce some nice ideas (and DIPS has something similar) which is using the full AQL-style path for the name and potentially the compact, stringified representations of the values- there is value in these stringified forms beyond composition serialisation - they are useful in GDL etc. and documentation

For now we have probably parked these (I think) as part of the next REST release as this still needs a bit of work.

I do actually like the idea (as you were suggesting) of being able to mix and match the various contracted forms (and including the ability to drop back to RAW canonical json) but that would require something a bit more clever!!

The priority, for me and I think Erik, certainly is to get the ehrscape web template formats supported, then we take a fresh look at new ideas and if we are lucky something even better.

On the last call we also discussed the ehrscape INPUT format (which adds the ctx directives - these are not data items per-se but instructions that help simplify the population of various mandatory items throughout the rest of the ocpmosiiton

e.g ctx/time , unless overriden, ppopulates the context/start_time and every madatory time attribute e.g ACTION.time with the value supplied. This very nice as it removes a lot of boilerplate in the composition. However when the composition is retrieved it is always as the fully populated version so you get a mismatch between the json object sent and the one retrieved.

So while I do use the ctx blocks when training, I tend to suggest that people get up to speed not using ctx, as sooner or later they will are likely to encounter it e.g if working with a persistent composition which needs updated.

Structured or Flat - choose your poison. It’s a cognitive preference, or perhaps JS coders prefer structured because it maps cleanly into their world with out the need to handle the Flat ‘indexers’. As a local organisation or company it might be reasonable to have a specific policy but I don’t think we should express a preference as partof the spec as such.

[thomas.beale]

@sebastian.iancu

the format specs are in DEVELOPMENT state, and located here: Simplified Data Template (SDT)

Just to be even more confusing, I created this separate spec, on the principle of keeping things as simple as possible: Serial Data Formats.

This was intended as a spec to collect the bare minimum elements of these (somewhat variable) JSON template formats, in the shortest form possible - and the contents should directly reflect what is in real products, and what is being used.

That SDT spec contains a (mainly my) more complex engineering approach to solving this, which may be over the top. I suggest the SEC decide which of these two specs to designate as the one we are actively working on.