Correct format for /composer/external_ref in FLAT compositions

Dileep_V_S · 10 September 2021 11:46

EHRBase 0.17.2 gives the following error if /composer/external_ref not given in FLAT compositions.
"error": "java.lang.IllegalArgumentException: Message at /namespace (/composer/external_ref/namespace): Attribute namespace of class PARTY_REF does not match existence 1..1\nMessage at /scheme (/composer/external_ref/id/scheme): Attribute scheme of class GENERIC_ID does not match existence 1..1\nMessage at /value (/composer/external_ref/id/value): Attribute value of class GENERIC_ID does not match existence 1..1\nMessage at /type (/composer/external_ref/type): Attribute type of class PARTY_REF does not match existence 1..1\n",

Tried different syntaxes, but could not get it to work, without success. Can anyone give the correct syntax?

"vital_signs/composer/external_ref/id|type": "GENERIC_ID",
"vital_signs/composer/external_ref/id|value": "123456",
"vital_signs/composer/external_ref/id|scheme": "userUuid",
"vital_signs/composer/external_ref/namespace": "EHR.NETWORK", 
"vital_signs/composer/external_ref/type": "ANY",

regards

birger.haarbrandt · 10 September 2021 12:56

Hi Dileep,

for the case this error cannot solved here by providing a matching syntax, please feel free to open a ticket in github.

Best,

Birger

Dileep_V_S · 10 September 2021 15:50

Thanks,

I also felt this was a bug and already raised a ticket😁

regards

ian.mcnicoll · 10 September 2021 16:58

Hi Dileep,

I have not tested this but the normal way of populating externalRef in Better structured (according to their docs) is

“composer”: [
{
“|id”: “77756”,
“|id_scheme”: “HOSPITAL-NS”,
“|id_namespace”: “HOSPITAL-NS”,
“|name”: “Silvia Blake”
}
]
or flattened equivalent

birger.haarbrandt · 10 September 2021 19:35

We currently supported the structured parts. Will try to have these as well sonly

Dileep_V_S · 12 September 2021 07:04

Thanks Ian,
In 16.5, we could commit FLAT with just the composer name. It was not accepting the external_ref values. In 17.2, it is rejecting the same composition and complains about the missing external_ref values. But it does not accept any format that I tried.

So I am kind of stuck and unable to progress further.

regards

jake.smolka · 13 September 2021 07:58

I’m a bit confused, because those log entries are corresponding to the new Archie invariant checks, which are now implemented into EHRbase. But this is (or at least should) not the case as of version 0.17.2.

If you are instead using the latest up-to-date docker image (called next) this feature would be already available indeed. In that case the readme tries to warn about this and the documentation has a config entry to disable the validation. For instance, in the form of an env var: SERVER_DISABLESTRICTVALIDATION=true.
Of course, caution is advised when using this option and further data input can’t be guaranteed to be correct. But this should help resolving this blocker for you, for now.

Dileep_V_S · 13 September 2021 09:38

Thanks Jake,
I have pulled ehrbase:latest image(pulled the latest code and used the docker compose file in that). So Archie invariant checks are probably included. Does that introduce any format changes in the FLAT format for external_ref?

Instead of bypassing this check, I would rather try and get the format right. Would you, by any chance, know the correct format?

regards

jake.smolka · 13 September 2021 11:33

I just manually checked it and there appears to be a bug in the internal transformation of the FLAT, so the external ref of the composer is not really piped through. There wouldn’t be anything you can do about it on the input level.
Let’s continue to add details to the issue on Github and it hopefully it will be addressed soon

ian.mcnicoll · 13 September 2021 14:31

for Better this works

 "annual_review/composer|name": "Dr Smith",
    "annual_review/composer|id": "12345",
    "annual_review/composer|id_scheme": "scheme",
    "annual_review/composer|id_namespace": "namespace"

and creates

   "composer": {
        "_type": "PARTY_IDENTIFIED",
        "external_ref": {
            "_type": "PARTY_REF",
            "id": {
                "_type": "GENERIC_ID",
                "value": "12345",
                "scheme": "scheme"
            },
            "namespace": "namespace",
            "type": "ANY"
        },
        "name": "Dr Smith"
    },

CzarMich · 14 August 2023 13:35

Any status update on this?
I am using Better but this on the patient side is too affected. openEHR Flat format does not contain externally referenced patients.
I used a TDD containing the segment below to create a composition. But when I request for a composition in flat format, the patient segment is never there

<subject type="oe:PARTY_SELF">
	<external_ref>
		<id>
			<value>YXXSSXXXXXXID</value>
			<scheme>PATIENT</scheme>
			<_type>GENERIC_ID</_type>
		</id>
		<namespace>1.2.3.4.5.6.7.8.9.200</namespace>
		<type>PERSON</type>
	</external_ref>
</subject>

I have attached a deducted composition in FLAT format out for this.

SampleFLATFomrmat.json (3.0 KB)

ian.mcnicoll · 16 August 2023 08:03

Ah!! - I think I figured this out.

There is actually no ‘subject’ attribute in the Composition class, by design. When a composition is created it is contained with a specific EHR object - the subject structure in the TDD is folding together the EHR.EHR_STATUS .

What you have created is an ENTRY.subject attribute which is fine, except that you have defined it as PARTY_SELF - which is the default, meaning that the information in the ENTRY refers to the person identified in the parent EHR. In this case the external_ref attributes will be ignored, which is why you are not seeing anything in the FLAT_JSON, which does not normally expose the subject attribute, unless it is set to something other than PARTY_SELF

Very rarely the information in an ENTRY refers to someone esle e.g. a family member or a foetus in a maternal record - and this where you might set te subject type as e.g PARTY_REF or PARTY_IDENTIFIED, and then the external_ref should appear in the FLAT_JSON.

So the short answer is that the normally the subject deatails are held outside the composiiton but you can retreive that via AQL, although I think it is also returned in FLAT_JSON metadata.

CzarMich · 16 August 2023 12:37

I tried to experiment with both “PARTY_IDENTIFIED” and “PARTY_REF” and also removing the attribute, but result in the FLAT_JSON is still the same; without the patient identifier.

Via AQL , it works as expected via the path e/ehr_status/subject/external_ref/id/value regardless of the Subject_type attribute.

 <subject type="oe:PARTY_IDENTIFIED">
            <external_ref>
                <id>
                    <value>dWjspnQmmRU6nguc9n2EFp</value>
                    <scheme>PATIENT</scheme>
                    <_type>GENERIC_ID</_type>
                </id>
                <namespace>1.2.2.3.4.5.6.7.8.9.200</namespace>
                <type>PERSON</type>
            </external_ref>
        </subject>

        <subject>
            <external_ref>
                <id>
                    <value>dWjspnQmmRU6nguc9n2EFp</value>
                    <scheme>PATIENT</scheme>
                    <_type>GENERIC_ID</_type>
                </id>
                <namespace>1.2.276.0.76.3.1.454.1.11.100</namespace>
                <type>PERSON</type>
            </external_ref>
            <_type>PARTY_IDENTIFIED</_type>
        </subject>

ian.mcnicoll · 16 August 2023 13:08

Can you show the wider context of where the subject attribute is being passed in the TDD? Is this part of an Entry level object?

CzarMich · 16 August 2023 13:23

Hi Ian,

I have attached the TDD used for the composition creation. And yes, the snippet is part of the entry level objekt.

Thank you.
Output.xml (7.4 KB)

ian.mcnicoll · 16 August 2023 18:16

So I’m not sure why the references are not appearing in the FLAT JSON, however they really are not needed, indeed you would be discouraged from carrying the sibject ID on each Entry.

The normal policy is just to apply PARTY_SELF which essentially says ‘use the parent subjectId details’ carried in the parent EHR object.

Out of interest, you might try changing

<type>PERSON</type>`

to

<type>PARTY_REF</type>

Though I’d still suggest that you just need to set just PARTY_SELF and do not try to include the subjectID for every ENTRY.

Also just for info, we actually find that FLAT_JSON makes a great target for xml or json integrations. We have not used TDD for a long time.

CzarMich · 16 August 2023 19:52

Unfortunately trying to change type to ‘PARTY_REF’ did not do any magic either.
<type>PARTY_REF</type>

TDD has been our main basis of work and not FLAT_JSON as we’ve mainly been mapping data from other sources into the openEHR CDR.

The requirement for FLAT_JSON arose from the need to map openEHR to FHIR. The tool I am using requires a composition in FLAT_JSON format. While an AQL can be used for this purpose, I wanted to first explore all possibilities before definitively declining the tool’s usage due to the lack of the Subject ID or EHR_ID in the FLAT_JSON .

ian.mcnicoll · 17 August 2023 16:22

Just to be clear, Michael - you do not need to record the subjectID/EHRId in each composition Entry. Just set the subject type to PARTY_SELF and do not add the external_ref. When the composition is created via the TD, you will be asked to supply the appropriate EhrID which will allocate it to the correct EHR object from which you can retrieve the subjectID/ehrId if necessary.

I’m not sure why you are not able to set PARTY_REF, along with an External_ref but this is not what you want or need to be doing.

Just do this

<subject type="oe:PARTY_SELF"/>

CzarMich · 17 August 2023 21:52

Hi Ian,

Unfortunately our pipeline requires the subject ID in the TDD.

We have a stream of 1000s of HL7v2 messages from different subjects coming from different sources daily or weekly etc. By all means I would have to supply the subject id from the HL7v2 into the TDD in order for the message to be mapped and the right composition created in the right EHR , but also registered in the XDS registry. This enables us to do away with that 2nd step you mention of having to supply the EHRID. And if a message comes with a subject ID that has no EHRID, the MPI automatically handles this, provided the Subject ID is known.
Looks like we are moving away form the real reason for this thread though.

Here is a simplified view of our pipeline.

sebastian.iancu · 18 August 2023 09:17

I am not sure I understand correctly all the requirements and design choices you have here, but in openEHR the COMPOSITIONs are always “stored” (contained) in an EHR, therefore you need to commit data in 2 steps: one to select the right context (the right EHR and its id, based on subjectId, etc) and the to post a composition resource. That is also the reason to can use PARTY_SELF, as there is always a “container” (i.e. the EHR) associated with a patient/PARTY around all compositions. The ehr_id in turn is already present in url of the composition, so need conceptually to add it also to composition’s content.