Which AOM2 release are AWB/Nedap using for exporting OPT2 to JSON?

borut.jures · 18 November 2021 13:34

Are they using the latest AOM2 release?

The release 2.3.0 has attributes marked as mandatory and they are missing in the JSON I get from AWB:

OPERATIONAL_TEMPLATE
- terminology_extracts (this is optional in the latest release)

But even the latest release has some attributes marked as mandatory that are missing in the JSON:

AUTHORED_ARCHETYPE
- build_id
- other_meta_data
ARCHETYPE
- is_differential

Nedap’s VSCode extension generates OPT2 JSON that has some extra properties (they doesn’t bother me) but uses different names for boolean properties (I have renamed them in my test files):

mandatory (instead of is_mandatory)
specialized (instead of is_specialized)
is_generated does use is_ prefix
build_uid is missing
description.details.language.uri is missing (it is marked as optional in the latest BASE release but not in the openehr_base_110.bmm)

Conclussions:

I believe AWB and Nedap’s VSCode extension are using the latest releases for AOM2 (and BASE). Can you confirm @thomas.beale @pieterbos ?

I don’t know how much time it takes to prepare AOM2 BMM for the “latest” release but if @thomas.beale is willing to do it I’ll test it for all the little inconsistencies like I did with 2.3.0 (most are found by the linter while it analyses the generated code)

Since I generate everything from the BMMs (no manual editing of the source code) I can test many different AOM releases in parallel without extra work. I would be happy to test OPT2 JSON for AOM 2.3.0 if somebody can send them to me (or any other AOM release since 2.0.6).

Same goes for Task Planner if I can get OPT2 JSON files for that.

borut.jures · 18 November 2021 19:21

Maybe it is not the “latest” AOM2 but the “first”:

  "adl_version" : "2.0.6",
  "rm_release" : "1.0.4",

I guess “adl_version” is the same as AOM release? It looks so based on the specifications.

pieterbos · 19 November 2021 15:58

VS code tool uses ADL 2.1.0, perhaps with some 2.2 features already.

Archie master branch on GitHub is 2.2.0, should contain nearly all 2.3.0 features. Note that 2.3.0 has not been released yet, 2.2.0 nearly (?). I will update ADL version numbers in the Archie output later…

Those Json inconsistencies, the missing is_ plus a bit more, need to be addressed. Probably configurable to ease upgrading. Now that we have a 2.3.0 BMM, I can run the test that compares it with an implementation and see if I missed more. I hope to have a bit more time to address them and release the next major Archie version soon.

Terminology extracts being mandatory, not sure if that should be the case. The Archie opt-creator never fills them in its current version.
I will check build Id and language URI later, could very well be that they should not be mandatory.

thomas.beale · 19 November 2021 16:25

Right - I am still adjusting that BMM.

It is (here), but if we find errors, they will get fixed on the branch. Any structural error will then get fixed in the BMM.

Seems that that is an error in all the BMMs. I’ll fix that now.

borut.jures · 20 November 2021 10:38

My notes on learning to parse OPT2 JSON files.

I used Nedap’s JSON for “openEHR-EHR-OBSERVATION.blood_pressure.v2.0.8_opt.json”:

“adl_version” : “2.0.6”,
“rm_release” : “1.0.4”,
“build_uid” : “54dfbb26-2c00-4001-b720-e0014a3dfb0f”,

Specifications:

Skip to later post for the list of changes to BMM files and JSON

I had to change the BMMs for the following properties:

original_language
- “Terminology_code” in openehr_base_104.bmm has “uri” as “is_mandatory=true”. Changed it to “false”.
- Latest online version has “uri” as optional.
archetype_id
- “ARCHETYPE_HRID” in openehr_base_110.bmm has “namespace” as “is_mandatory=true”. Changed it to “false”.
- Online version has “namespace” as optional.

I had to rename the following properties:

generated => is_generated
differential => is_differential

I had to add the following properties:

terminology
- “ARCHETYPE_TERMINOLOGY” in openehr_base_110.bmm has “is_differential” as mandatory. Added property.
- Online version has “is_differential” as mandatory.
terminology.term_definitions
- “ARCHETYPE_TERM” in openEHR_am_206.bmm has “code” as mandatory. Added property:
  - Search:
```
  ("[a-z]+\d+") : \{
    "@type" : "ARCHETYPE_TERM",      
```
  - Replace:
```
  $1 : \{
    "@type" : "ARCHETYPE_TERM",
    "code" : $1,
```
- Online version has “code” as mandatory.
- “ARCHETYPE_TERM” in openEHR_am_206.bmm doesn’t have “text”, “description”, “other_items” properties.

Edit: Please skip the rest. I forgot that this is only a template and not the “OBSERVATION” data that is sent to the CDR.

I was stopped at the property “definition”:

“OBSERVATION” in openehr_rm_ehr_104.bmm has “data” as mandatory.
Online version has “data” as mandatory.
JSON has “data” as part of its “attributes” of a “C_COMPLEX_OBJECT”
I was expecting that “data” will be part of the “definition” property. It is instead inside its “C_COMPLEX_OBJECT” attributes. This is different as other models above.
We know that “definition” has “OBSERVATION” from its “rm_type_name” : “OBSERVATION” property.
I was expecting to use the following code to read “definition” property as “OBSERVATION” from JSON (the same concept as for e.g. “ARCHETYPE_TERMINOLOGY”, “Terminology_code” above):

Edit: “@type” is not needed after all since the “ARCHETYPE.definition” is of type “C_COMPLEX_OBJECT”. The following two lines may be skipped:
Edit2: “@type” is needed for the “attributes.children” properties.

This also means that the “@type” property in Nedap’s JSON is needed.
AWB’s JSON doesn’t have “@type” property to indicate what descendant “C_OBJECT” type is an element of “attributes.children”.

I’ll have to read “OBSERVATION” from the “C_COMPLEX_OBJECT” attributes and not as a first class object with its own properties.

I’ll search for the explanation in the specifications why “OBSERVATION” is stored differently compared to e.g. “ARCHETYPE_TERMINOLOGY”.

borut.jures · 20 November 2021 11:02

Maybe to explain a bit more about my approach.

“OBSERVATIONAL_TEMPLATE” determines a type of its “definition” property at runtime:

The “fromJsonRuntime()” method:

The “_runtimeObjectFactories” that specify which object’s fromJson() method is called based on the “rm_type_name” property in the JSON:

Edit:
I checked my generator for the “OBSERVATIONAL_TEMPLATE.definition” property. It skips the “C_COMPLEX_OBJECT.fromJson()” and goes directly to the runtime type specified by the “rm_type_name”.

I have to first read “C_COMPLEX_OBJECT.fromJson()” which will then call “OBSERVATION.fromJson()” based on the “rm_type_name”.

But I’ll probably need “OBSERVATION.fromCComplexObjectAttributes()” since the “OBSERVATION” data is inside “C_COMPLEX_OBJECT.attributes” and not in the direct JSON form specified by “OBSERVATION”.

borut.jures · 21 November 2021 19:28

I can read my first OPT2 JSON

…and I have to write only a single line of code to do it:

…everything else is generated from the BMM files.

When I started working with openEHR I hoped this could be done. 40 days later I know it can be done. All thanks to the computable specifications.

I’m especially thankful to @thomas.beale and @pieterbos for their help.

All the changes to the BMMs and JSON I had to do:

Changed BMMs:

original_language
- “Terminology_code” in openehr_base_104.bmm has “uri” as “is_mandatory=true”. Changed it to “false”.
- Latest online version has “uri” as optional.
archetype_id
- “ARCHETYPE_HRID” in openehr_base_110.bmm has “namespace” as “is_mandatory=true”. Changed it to “false”.
- Online version has “namespace” as optional.
constraint
- “C_TERMINOLOGY_CODE” in openEHR_am_206.bmm has "constraint.type = <“String”>.
- “C_TERMINOLOGY_CODE” in openEHR_aom_206-generated.bmm (generated from Archie) has "constraint.type_def = (P_BMM_CONTAINER_TYPE).
- Online version has "constraint.type = <“String”>.
- “C_STRING” in openEHR_am_206.bmm has “constraint.type” = P_BMM_CONTAINER_PROPERTY.
- I changed “C_TERMINOLOGY_CODE.constraint.type” in openEHR_am_206.bmm to P_BMM_CONTAINER_PROPERTY like it is already used for “C_STRING”.
“Iso8601_type”
- “value” is missing “is_mandatory = True” in openehr_base_104.bmm and openehr_base_110.bmm.
- Online version has “value” as mandatory.

Renamed properties in JSON:

generated => is_generated
differential => is_differential
multiple => is_multiple

Added properties in JSON:

terminology.is_differential
- “ARCHETYPE_TERMINOLOGY” in openEHR_am_206.bmm has “is_differential” as mandatory. Added property “terminology.is_diffrential” to the JSON.
- Online version has “is_differential” as mandatory.
terminology.term_definitions
- “ARCHETYPE_TERM” in openEHR_am_206.bmm has “code” as mandatory. Added the “terminology.term_definitions.code” property to the JSON.
  - Search:
```
  ("[a-z]+\d+") : \{
    "@type" : "ARCHETYPE_TERM",      
```
  - Replace:
```
  $1 : \{
    "@type" : "ARCHETYPE_TERM",
    "code" : $1,
```
- Online version has “code” as mandatory.
- “ARCHETYPE_TERM” in openEHR_am_206.bmm doesn’t have “text”, “description”, “other_items” properties.

Unnecessary properties (I skipped them):

“attributes” ("@type" : “C_ATTRIBUTE”) properties have a “mandatory” which is not part of the specification.
“occurrences” ("@type" : “MULTIPLICITY_INTERVAL”) properties have a “mandatory” which is not part of the specification.

Now I can continue with a forms entry application based on an OPT

Should I generate it from BMMs?

borut.jures · 23 November 2021 18:34

Thank you for removing is_mandatory for “terminology_extracts”.

I missed one more in the original post:

component_terminologies are marked as mandatory in BMMs, but not in the specifications:
- AOM 2.0.6
- AOM latest

thomas.beale · 23 November 2021 18:59

This is now fixed in the BMMs. Thanks for reporting it.

borut.jures · 23 November 2021 20:35

One more but this one might be an error in the specifications (or my understanding of them).

“other_meta_data” is marked as mandatory in the specifications and in the BMM (so BMM and specifications are in-sync):

Nedap’s JSON has this as an empty map property:

"other_meta_data" : { },

This satisfies the validator but I (and my code) read the specifications as: “there must be at least one other_meta_data element”.

There is no need for the other meta data on some templates.

Shouldn’t a HashMap property (like “other_meta_data”) that can be empty, be marked as is_mandatory = False?

Or should I change the code to allow mandatory but empty hash map properties?

borut.jures · 24 November 2021 09:45

@thomas.beale It looks like you missed this one in my long notes on AM 206 BMM. Please update the official BMM.

AOM 2.0.6

thomas.beale · 24 November 2021 10:26

Also done now…!

pieterbos · 24 November 2021 13:06

One more: The expressions bmm does not include the base bmm. So the AOM bmm file fails to validate because it cannot find String

pieterbos · 24 November 2021 13:07

Oh, and CONSTRAINT_STATUS is not part of a package, so that also fails to validate.

borut.jures · 24 November 2021 13:18

I added an include to my local AM 206 BMM and also removed few classes:

-- 2021-11-15; bj; Removed duplicate classes found in openehr_base_110.
--		classes = <"AUTHORED_RESOURCE", "TRANSLATION_DETAILS", "RESOURCE_DESCRIPTION", "RESOURCE_DESCRIPTION_ITEM", "RESOURCE_ANNOTATIONS">

-- 2021-11-15; bj; Removed duplicate classes found in openehr_base_110.
				classes = <"C_COMPLEX_OBJECT", "C_PRIMITIVE_OBJECT", "C_PRIMITIVE_TUPLE", "C_SECOND_ORDER",
					"C_ATTRIBUTE_TUPLE", "C_ATTRIBUTE", "ARCHETYPE_SLOT", "C_OBJECT", "C_COMPLEX_OBJECT_PROXY", "C_DEFINED_OBJECT",
--					"SIBLING_ORDER", "C_ARCHETYPE_ROOT", "ARCHETYPE_CONSTRAINT",
--					"Cardinality", "Multiplicity_interval"
					"SIBLING_ORDER", "C_ARCHETYPE_ROOT", "ARCHETYPE_CONSTRAINT"
        >

thomas.beale · 24 November 2021 15:05

Well the AOM includes the base bmm directly anyway, so that wouldn’t be right. But I see that the inclusions of base and expressions were around the wrong way - it should have been Expression including Base, not the other way round. This is now fixed, and should validate in Archie & @borut.jures 's framework as well.

Yep - I have not finalised 2.3.0 in fact…

What you are doing here manually is what the ADL Workbench does internally with a bit more code. This is generally necessary, since in a graph of inclusions from some high-level model, things like base inevitably get included by more than one node in the graph. The only way to really do this properly is to process the separate BMMs into memory and handle duplicates there; trying to process the serial form into a equivalent single serial
without going via the in-memory processing is pretty well impossible - you can’t avoid doing the logic of the in-memory processing. I probably should re-engineer that piece of code from ADL Workbench into Java some day…

pieterbos · 24 November 2021 15:25

It validates expressions separately as well, which does not validate because of missing includes. It then tries to include a schema that did not pass validation, which is a separate error in itself, so AOM will fail to validate as well in Archie, unless the include is added of course.

borut.jures · 24 November 2021 15:47

Thank you.

One change is missing in the “openehr_base_104.bmm” and “openehr_base_110.bmm” and then I can use the standard ones:

And another in “openehr_base_110.bmm”:

“openEHR_am_206.bmm” and later versions also depend on “openehr_expression_104.bmm” but use “openehr_base_110.bmm” instead of “openehr_base_104.bmm”.

I created a copy of “openehr_expression_104.bmm” that includes 110 instead of 104. Maybe there should be “openehr_expression_110.bmm” (110).

I hope the last fix for the “openEHR_am_206.bmm”:

“TERMINOLOGY_RELATION” is missing “members” property in “openEHR_am_206.bmm”:
AM 2.0.6

pieterbos · 24 November 2021 15:58

Did an automated comparison. Results so far, apart from the missing is_ prefixes in Archie:

version_last_translated is missing from AUTHORED_RESOURCE, both in spec and BMM, but present in AOM documentation and in Archie:
https://specifications.openehr.org/releases/AM/latest/AOM2.html#_version_last_translated

RESOURCE_DESCRIPTION_ITEM.copyright is missing in Resource Model

cardinality.is_ordered and is_unique should be mandatory, isn’t in BMM

RESOURCE_DESCRIPTION_ITEM.other_details

C_SECOND_ORDER.members is not mandatory in spec, but is in BMM. Is mandatory in subclasses though, should probably be mandatory?

It’s possible that I will find more later, enough for today.

thomas.beale · 24 November 2021 18:45

Fixed.

Fixed

That is actually correct - the Expressions uses the related version of base (1.0.4). You can no doubt get away with allowing it to be base 1.1.0, but formally that is not correct - by BASE Release 1.1.0, that Expressions model is gone.

Fixed.

It is actually in TRANSLATION_DETAILS; but was missing from the BMMs. Fixed.

It is in RESOURCE_DESCRIPTION class; the BMM had an erroneous duplicate in RESOURCE_DESCRIPTION_ITEM, now removed.

It was not in the Base 1.1.0 BMM; fixed.

The error here was that the AOM2.0.6 and the BASE 1.1.0 BMMs had it as mandatory instead of optional; fixed.

Yes - the BMMs are correct here. The UML is not quite correct on this.

Not a bad haul for today