# Create native JSON composition format from OPT **Category:** [Platform](https://discourse.openehr.org/c/platform-implem/7) **Created:** 2019-11-29 11:02 UTC **Views:** 5473 **Replies:** 71 **URL:** https://discourse.openehr.org/t/create-native-json-composition-format-from-opt/198 --- ## Post #1 by @Dileep_V_S Hi, I am trying to test the EHRBase APIs and have been able to create EHR and sample templates(did not return success though the template was created) . The next is to create compositions. Here I am stuck as the API expects the native JSON format for the composition. How can I create the JSON composition format from OPT? Is there any tool that I can use? In Ethercis, I used the template example format API to get a sample that I can use for compositions, but could not find something similar in EHRBase. Also is the git repository the preferred location to log bugs ? regards --- ## Post #2 by @ian.mcnicoll My suggestion (for now) is to create the template composition example in Ethercis, save that composition, Get the composition in RAW JSON format and bingo, you have a JSON RAW format composition. Might be quite easy to string together with a wee bit of code. I would be pretty sure that Github is the preferred option for reporting bugs Ian --- ## Post #3 by @backbord EHRbase does also accept XML. The only part that is restricted to JSON is AQL. Also EHRbase has a client library with recently made some progress: https://github.com/ehrbase/ehrbase_client_library I started off with generating XML instances here: https://server001.cloudehrserver.com/cot/opt/xml_instance_generator Right now it looks like the certificate is expired, but it still works. Just make sure to select "Composition" in the Options. Then upload the composition to the EHRbase Server with the following headers to get it back in JSON format: Prefer: return=representation Accept: application/json Just make sure that your build of the EHRbase server is at least commit 638a0fb if your template contains clusters. Best regards Niklas --- ## Post #4 by @pablo Hi Niklas, [quote="backbord, post:3, topic:198"] I started off with generating XML instances here: https://server001.cloudehrserver.com/cot/opt/xml_instance_generator [/quote] I'm working to make the CaboLabs openEHR Toolkit to generate canonical JSON instances directly, since the current JSON instances we generate are just a transformation from the canonical XML to JSON It seems I need to update the certs :) good catch! --- ## Post #5 by @erik.sundvall Do I understand correctly that CaboLabs openEHR Toolkit to generate canonical JSON examples directly from an OPT-file? EDIT: might have found it at https://server001.cloudehrserver.com/cot/opt/json_instance_generator will test... --- ## Post #6 by @erik.sundvall Is there anything out there like https://github.com/ehrbase/ehrbase_client_library but for Javascript/Typescript instead? Something that exposes the RM or generate template-specific classes to client javacript code? --- ## Post #7 by @ian.mcnicoll Not yet but I want it!! --- ## Post #8 by @birger.haarbrandt @erik.sundvall it's not as convenient, but it should be possible to convert the JPA like classes to a simplified JSON format, allowing to build form specific REST interfaces. I know that this is not the same as directly setting values in the class but could be a nice method while we are waiting for such a tool to occur :wink: --- ## Post #9 by @erik.sundvall Ok, I agree with @ian.mcnicoll, so let's figure out a simple way to get started, and then a way to do it properly. I mixed two tings in one post, i'll try to split and name the two... Project **RM2TS**: from RM BMM-files, JSON-schema etc via some tooling/code to Typescript classes with generic RM names * I bet somebody has already set up a toolchain that can parse BMM (https://github.com/openEHR/specifications-ITS-BMM) and then output RM-objects in their favourite programming language using some templating language or configurable generator that could be modified to output Typescript classes instead. If they can/want to share that toolchain or not is of course a question. * If you're not @thomas.beale and building a BMM-parser is a favourite hobby, then BMM can be exported in JSON format from Tom's magic ADL Workbench http://openehr.github.io/adl-tools/adl_workbench_guide.html An (old) example of such an export is available at https://github.com/regionostergotland/openehr_definitions/blob/master/rm/1.0.4/openehr_rm_1.0.4.json * But on the other hand there is likely some general tool that can take the openEHR JSON schemas at https://github.com/openEHR/specifications-ITS-JSON and turn into beatiful TypeScript hopefully with serializer & deserializer if needed... Any suggestions? @sebastian.iancu what did you use to generate those schemas? Project **OPT2TS**: from operational templates (OPT) via some tooling/code to template-specific Typescript classes with use-case-meaningful names * @birger.haarbrandt I assume by JPA you mean https://en.wikipedia.org/wiki/Java_Persistence_API right? And that you are thinking of template-specific simplified models? * Services like the https://rest.ehrscape.com/rest/v1/template/{templateId}/example described at https://www.ehrscape.com/api-explorer.html seem capable of producing somewhat readable structures in JSON if you pick format="STRUCTURED" when calling it. That could maybe be fed to services turning JSON examples to Typescript classes... * ...but generating Typescript from the OPT somehow would likely be a lot safer and cleaner --- ## Post #10 by @thomas.beale For RM2TS or RM2xyz, be aware that [BMM has been upgraded significantly](https://specifications.openehr.org/releases/LANG/latest/bmm.html#_expressions) to cover pretty much all semantics of normal specification languages, i.e. the same semantics of most programming languages at interface level, and expressions. The only thing missing (I don't think it's needed right now) is procedural control structure statements like if/then/else etc. Some of the most recent changes: * support for agents (aka lambdas), as calls, delayed calls, and 'signature' types * fully described routine definitions, including pre- and post-conditions * class invariants * better meta-data and documentation structures Of course, these still need to be transferred into projects like Archie (@pieterbos - be afraid, be very afraid ;) but the general direction is that the primary definition of all openEHR information models (RM, BASE, TP etc) will be in BMM in sufficient precision to enable generation of any code interfaces in any reasonable OO+FP language. --- ## Post #11 by @yampeku If there is a tool that can be used to generate JSON data from a JSON schema, I added JSON schema generation to LinkEHR some time ago, maybe it's enough for your use case. I think we are a little behind BMM latest changes though... :sweat_smile: :sweat_smile: --- ## Post #12 by @erik.sundvall Interesting. Thanks! There are things like https://github.com/bcherny/json-schema-to-typescript#readme (demo at https://bcherny.github.io/json-schema-to-typescript-browser/ ) and https://quicktype.io/typescript/ and several others. A drawback is likely that the route via json schema likely removes some detail from the BMM that typescript (or extension classes/libraries) could handle better than JSON can. I am guessing a bit, so I could be wrong, https://spin.atomicobject.com/2018/03/26/typescript-data-validation/ that describes this approach says "JSON schema actually provides a superset of the basic features of the TypeScript type system for runtime validation". Modifying the pipeline in your tool to outputting typescript directly (instead of just json schema) from BMM might be even more interesting. Is that part open sourced? @yampeku could you post a link to the part/classes/code that does the conversion? --- ## Post #13 by @sebastian.iancu [quote="erik.sundvall, post:9, topic:198"] @sebastian.iancu what did you use to generate those schemas? [/quote] Those are generated using of min PHP code from the UML .xmi files we have. However, those schemas still needs some work to make them more accurate, to remove annotations, and potentially to be generated from BMM instead of UML. --- ## Post #14 by @yampeku [quote="erik.sundvall, post:12, topic:198"] Modifying the pipeline in your tool to outputting typescript directly (instead of just json schema) from BMM might be even more interesting. Is that part open sourced? @yampeku could you post a link to the part/classes/code that does the conversion? [/quote] It's not from BMM directly, but from any given archetype (which can be derived from RM archetypes based on BMM). This functionality is fully available on LinkEHR, if it's from an end user perspective maybe easiest thing would be to download an use it from there. Code is not currently open, but I can liberate this part without problems. Has dependencies to the old java libs (not archie) and code is quite "handcrafted", but does the job. If you are interested I could clean it a little and publish it in a git repo or something. --- ## Post #15 by @erik.sundvall @yampeku don't put too much time into publishing that for our sake yet, let's explore some other options first. Or are your "RM archetypes" already auto generated from BMMs in some other pipeline? I'd like to find a (semi)automated way from BMM to Typescript/Javascript-library or some already actively maintained library. @bna I believe you mentioned the other day that DIPS already has some TypeScript library for openEHR that makes Intellisense etc in IDEs work. When looking at https://github.com/DIPSAS/ehrcraft-pre-procurement-demo/tree/master/typescript, the files in the "/src"-directory seem to import openEHR types such as `DvQuantity, DvBoolean, DvCodedText`. * Where are those classes imported from? * Are they available as open source (or could be made available)? * How did you create them - handcrafted, BMM-conversion or something else? --- ## Post #16 by @bna You are right @erik.sundvall. There is a Typescript library for the Form script types. It's not a real implementation just the API definitions to make dev of Form scripts faster. The package is published to NPM and should be available for anyone. https://www.npmjs.com/package/ehrcraft-form-api/v/1.0.1 --- ## Post #17 by @erik.sundvall Great! Thanks @bna! How did you create them - handcrafted, BMM-conversion or something else? --- ## Post #18 by @bna Handcrafted. It was a need using the platform to get intellisense building Form scripts. It was also a minor RD project to test if it was possible to do it like this. The hardest part was actually compiling TS into supported JS syntax without upgrading the Form Designer. This is done by some tricks in the Gulp script also added to the Github project. --- ## Post #19 by @yampeku [quote="erik.sundvall, post:15, topic:198"] Or are your “RM archetypes” already auto generated from BMMs in some other pipeline? [/quote] Yep, that's what linkehr uses for rm representation. RM is defined as archetypes with special atcodes. These are bundled with every linkehr, available in the data/rm folder. There you will find rm archetypes for openEHR, CDA, ISO13606 and even some FHIR STU3 Resources --- ## Post #20 by @surfer Hi all, Reconnecting to the first question of the post, do you have a couple template-composition that works in Ehrbase? (I'm on the latest, ehrbase-0.10.0) And can you explicit the header and parameter settings needed? I tried many couple, in xml, (that works with ThinkEHR and/or Ethercis) with no success. Then I tried the procedure proposed by @ian.mcnicoll , that is get in JSON a composition saved in Ethercis in XML, but the only format Ethercis can export to seems to be XML and ECISFLAT. I tried to insert the ECISFLAT in Ehrbase but I failed. I also created a composition from the cloudehrserver tool mentioned by @backbord. I tried both the xml and the json composition but it didn't work. Any help is appreciated :slight_smile: Thanks Giovanni --- ## Post #21 by @birger.haarbrandt Hi Giovanni, do you use the master branch? Could you please try the latest develop branch and check if the error still occurs? Cheers, Birger --- ## Post #22 by @ian.mcnicoll @surfer am pretty sure Ethercis does export in 'RAW' JSON - just use format=RAW as a parameter and change Content-type to application/json. I can't test so apols if I got that wrong --- ## Post #23 by @Dileep_V_S I can confirm what Ian mentioned about generating RAW compositions from Ethercis. However I have not yet tried to commit it to EHRBase. regards --- ## Post #24 by @surfer This is what I get in Ethercis if I put 'format':'RAW' in parameters, regardless of the Content-type chosen: htp://127.0.0.1:8080/rest/v1/composition/d5745655-f131-4856-a72d-6537d5c005a9::ethercis.crs4::1?format=RAW 500 Error 500 #1.0.0 Invocation exception to method:retrieve:java.lang.IllegalArgumentException: Invalid element detected in map

HTTP ERROR 500

Problem accessing /rest/v1/composition/d5745655-f131-4856-a72d-6537d5c005a9::ethercis.crs4::1. Reason: 

    #1.0.0 Invocation exception to method:retrieve:java.lang.IllegalArgumentException: Invalid element detected in map

Powered by Jetty:// 9.4.z-SNAPSHOT
The same composition with 'format':'ECISFLAT': htp://127.0.0.1:8080/rest/v1/composition/d5745655-f131-4856-a72d-6537d5c005a9::ethercis.crs4::1?format=ECISFLAT 200 { "composition" : { "/composer|name" : "Physician", "/content[openEHR-EHR-ACTION.procedure-sus.v1]/description[at0001]/items[at0002]|value" : "SUS::0408060140|FASCIECTOMIA|", "/content[openEHR-EHR-ACTION.procedure-sus.v1]/ism_transition/careflow_step|value" : "local::at0043|Completed|", "/content[openEHR-EHR-ACTION.procedure-sus.v1]/ism_transition/current_state|value" : "openehr::532|completed|", "/content[openEHR-EHR-ACTION.procedure-sus.v1]/time|value" : "2018-12-11T11:20:07,230", "/content[openEHR-EHR-ADMIN_ENTRY.admission.v1]/data[at0001]/items[at0013]|value" : "local::at0020|Eletiva|", "/content[openEHR-EHR-ADMIN_ENTRY.admission.v1]/data[at0001]/items[at0041]|value" : "Cirúrgico", "/content[openEHR-EHR-ADMIN_ENTRY.admission.v1]/data[at0001]/items[at0071]|value" : "2010-09-01T00:00:00-03:00", "/content[openEHR-EHR-ADMIN_ENTRY.admission.v1]/data[at0001]/items[at0073]/items[at0084]/items[at0087]|value" : "IBGE::MT|MATO GROSSO|", "/content[openEHR-EHR-ADMIN_ENTRY.admission.v1]/data[at0001]/items[at0073]/items[at0104]/items[at0106]|value" : "CGC_HOSP::029f40596e8f365e39a10337202fcde4|4dc5277c8fb69ad41181cad1844491e6d2eb7999ae9f638a|", "/content[openEHR-EHR-ADMIN_ENTRY.hospitalization_authorization.v1]/data[at0001]/items[at0016]|value" : "0", "/content[openEHR-EHR-ADMIN_ENTRY.hospitalization_authorization.v1]/data[at0001]/items[at0061]|value" : "2010-09-01T00:00:00-03:00", "/content[openEHR-EHR-ADMIN_ENTRY.patient_discharge.v1]/data[at0001]/items[at0002]|value" : "2010-09-01T00:00:00-03:00", "/content[openEHR-EHR-ADMIN_ENTRY.patient_discharge.v1]/data[at0001]/items[at0047]|value" : "MOTSAIPE::12|Alta - melhorado|", "/content[openEHR-EHR-ADMIN_ENTRY.patient_discharge.v1]/data[at0001]/items[at0048]|value" : "false", "/content[openEHR-EHR-EVALUATION.problem_diagnosis-sus.v1]/data[at0001]/items[at0002.1]|value" : "CID10::M720|M72.0 Fibromatose de fascia palmar|", "/context/setting" : "openehr::232|secondary medical care|", "/context/start_time" : "2018-12-11T11:20:07.227+01:00", "/language" : "en", "/territory" : "BR" }, "meta" : { "href" : "rest/v1/composition?uid=d5745655-f131-4856-a72d-6537d5c005a9::ethercis.crs4::1" }, "format" : "ECISFLAT", "templateId" : "hospitalization_ocean" @Dileep_V_S how did you do it? --- ## Post #25 by @ian.mcnicoll Can you upload the .opt as well @surfer? I will try and test it later today. I would suggest you use the xml for now - I played with Better and ehrBase CDRs last night and got xml moving between the two with only as minor tweak to add the openehr schema to the namespace. THe raw json formats were somewhat different, which is not a surprise as this is still very early in adoption/ conformance testing. --- ## Post #26 by @surfer Thanks Ian, Here they are (composition and template): https://gist.github.com/sasurfer/5bd034a52af95b08d4ab59d1566c9fe5 --- ## Post #27 by @surfer Yes. I'm using the master branch. The zip 0.10.0 official alpha release https://github.com/ehrbase/ehrbase/archive/v0.10.0.tar.gz When I try with the develop branch I get this error: Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile (default-compile) on project test-data: Fatal error compiling: invalid flag: --release -> [Help 1] I'm trying to investigate the cause... --- ## Post #28 by @birger.haarbrandt The latest version on the dev branch requires Java 11. Maybe this is the issue. --- ## Post #29 by @surfer Thanks, It's definitely something I have to correct --- ## Post #30 by @surfer Built up the develop branch version successfully. I tried the composition and template posted but I receive the same error: 500 {"error":"Unable to create an instance of com.nedap.archie.rm.support.identification.UIDBasedId","status":"Internal Server Error"} Any idea of what's going wrong? --- ## Post #31 by @birger.haarbrandt Hi Giovanni, the first issue I found is regarding the namespaces in XML. The file should look like this: Hospitalisation openEHR-EHR-COMPOSITION.hospitalisation.v1 hospitalization_ocean 1.0.2 ISO_639-1 en ISO_3166-1 BR However, there seem to be more issues regarding the validation of DV_CODED_TEXT. We will take a closer look what's going on. I will hopefully have some news next week. One issue I was able to resolve was this one: Admission type Eletiva local at0020 as the language was set to "en", I had to replace "Eletiva" with "Elective". --- ## Post #32 by @ian.mcnicoll Hi Birger, That XML namespace issue is one I have known about - I think it is down to the slightly different behaviours of the serialization libs used by Better and ehrBase (or more likely somebody told me that!!). The issue about codedText validation of the text value as well as the code_string is interesting - it might be worth raising this with other implementers and @thomas.beale. I can see some arguments that validating on the text is not required or helpful. There are going to be some interesting issues in this area around SNOMED CT synoynms. --- ## Post #33 by @surfer thanks @birger.haarbrandt . looking forward to see the file cleaned. @ian.mcnicoll What I can tell, in my little experience, is that starting from the ORBDA db I had to make corrections, both in Template and Compositions, to use Better and Ethercis that aren't overlapping. In general Better is more "forgiving" and needs less corrections. And now Ehrbase is again different from the other two in terms of adjustments needed. --- ## Post #34 by @birger.haarbrandt At least the namespace handling is something that we could certainly fix by doing some internal pre-processing. However, I cannot say when this will be ready and for now this needs to be done client-side. --- ## Post #35 by @ian.mcnicoll It is something to consider as part of conformance testing before you 'fix' it - it is possible that it is Better who are doing things slightly differently. Or does this matter, at all in conformance terms. The Better approach makes sense, in their terms, where the element is not the root element - the openEHR ns is on the root which is a simple wrapper. These are just the small (hopefully!!) niggles that confomance testing will bring up and that we need clarify. --- ## Post #36 by @birger.haarbrandt I think from an XML perspective, the use of namespaces in the example given by @surfer looks appropriate. I think it still is canonical XML. Hence, I think EHRbase should accept the compositions. Birger --- ## Post #37 by @ian.mcnicoll digging into this a little further, I suspect it should not be an issue. It is I think an artefact of the Ehrscape GET /composition that wraps the composition in a metadata header (which carries the openEHR ns) . From what I can see the REST API mandates that the composition is the root element and therefore should carry the ns there. I'll test ASAP. --- ## Post #38 by @ian.mcnicoll Ah now I see I am talking nonsense!! This not the lack of an openEHR namespace on the root but the fact that it is not expressed as a 'default' namespace. I agree @birger.haarbrandt the format that @surfer sent over should be acceptable, as should the current ehrBase approach (digging back into my creaky knowledge of XML namespaces ). --- ## Post #39 by @matijap [quote="ian.mcnicoll, post:37, topic:198"] It is I think an artefact of the Ehrscape GET /composition that wraps the composition in a metadata header [/quote] I believe you've figured it all out now (did not check, but your last post makes sense), I'd just like to point out that Better EHR (and consequently Ehrscape) has 2 REST APIs; the legacy proprietary one, and the slightly different openEHR one. --- ## Post #40 by @ian.mcnicoll Yep - just doing a little tidyup of a single Postman collection that will handle both, so we can start to compare and contrast. --- ## Post #41 by @surfer Hello @birger.haarbrandt . Did you have any success in cleaning and submitting the composition I had problem with? Does anybody have a working pair template-composition from which I can start understanding what was wrong with my tests? --- ## Post #42 by @ian.mcnicoll You might want to keep[an eye here as we will be working between Ehrscape and ehrBase when building sample compositions. Sorry -a but busy with this to answer directly but some of the test material might help https://github.com/AppertaFoundation/COVID-19-screening-interface --- ## Post #43 by @birger.haarbrandt Hi @surfer, please the followin PR containing the analysis: https://github.com/ehrbase/ehrbase/pull/186 We had some issues regarding null flavours. This should also help you regarding the other aspects of the composition. --- ## Post #44 by @surfer thank you. I'm definitely gonna take a look at it as soon as my internet connection resurrects --- ## Post #45 by @surfer Hi again, Your link @birger.haarbrandt was precious to understand the errors in my xml composition. Now I'd like to be able to submit a json composition which, if I understood well, is the only one that allows to make AQL queries. Does somebody have a couple template/json composition even very simple that is accepted by Ehrbase? @ian.mcnicoll I looked at the files you linked (and not). I can see templates but no composition. I grabbed the NES-ACP_COVID.v0.0.opt and the openEHR-Suspected Covid-19 assessment.v0.opt I then got the json structured composition examples for those templates from Better API ( GET /template/{templateId}/example ) but they haven't been accepted in Ehrbase. --- ## Post #46 by @ian.mcnicoll Here the trick!! Use the Better Ehrscape API to create FLAT examples and then submit them to EBetter CDR via ehrscape /composition /POST - note the composiitonID THen ... us the Better **openEHR API** to retrive the composition via the compositonID as RAW JSON. Bingo you have an example RAW JSON which will commit to EhrBase. Have a look in the GH repo above for Postman examples and in the dev branch openEHR folder some examples. --- ## Post #49 by @surfer @ian.mcnicoll I just followed your instructions. It lead me to the same result that is : 400 {"error":"java.lang.IllegalArgumentException: Composition missing mandatory attribute: name","status":"Bad Request"} to summarize the steps I took: -I created a FLAT JSON composition from template example REST call in BETTER -I submitted to BETTER the same composition as FLAT JSON -I retrieved the composition from BETTER as RAW JSON -i tried to insert it in Ehrbase as RAW JSON --- ## Post #50 by @ian.mcnicoll Sorry too busy to try to replicate. > -I retrieved the composition from BETTER as RAW JSON The critical step here is that you retreive the RAW JSON via the Better openEHR API not via Ehrscape. Ehrscape generates 'BETTER RAW' openEHR API generates 'OPENEHR Canonical/RAW) They are actually very close. --- ## Post #51 by @ian.mcnicoll You might also try to recommit the CANONICAL RAW back to Better via the openEHR API - it sould work, then try it against EhrBase. --- ## Post #52 by @surfer I know @ian.mcnicoll I'm bothering you a lot but I can't make it work :( Which is which when you talk about EHRscape API and BETTER API ? Better has two API. One is called OpenEhr REST API whereas the latter , much larger , for the part we are referring to is called Electronic Health Record API. The first time you suggest me your trick I used only the second one. So i retrieved with API GET /composition/{uid} Last time I retrieved the composition with the first one (API GET /rest/openehr/v1/ehr/{ehr_id}/composition/{versioned_object_uid} ) The error with the second approach when posting the retrieved composition to Ehrbase is: 400 {"error":"java.lang.IllegalArgumentException: Composition missing mandatory attribute: archetype_node_id","status":"Bad Request"} --- ## Post #53 by @ian.mcnicoll Are you still using the same samples from the Gist you posted? --- ## Post #54 by @ian.mcnicoll > I retrieved the composition from BETTER as RAW JSON Yes but this has to be via the Better openEHR API not the Better Ehrscape API (Electronic Health Record API) curl --location --request GET 'https://cdr.code4health.org/rest/openehr/v1/ehr/3e674739-950c-4b8a-976b-5aef21c618c5/composition/89631516-6359-4b08-9500-b0a32b1e5d4f::a81f47c6-a757-4e34-b644-3ccc62b4a01c::1' \ --header 'Accept: application/json' \ --header 'Content-Type: application/json' \ --header 'Authorization: Basic YTgxZjQ3YzYtYTc1Ny00ZTM0LWI2NDQtM2NjYzYyYjRhMDFjOiQyYSQxMCQ2MTlraQ==' I will send over a postman collection that shows this working --- ## Post #55 by @surfer That's exactly what I did last time. Same parameters. looking forward to see your postman commands to see where I went wrong --- ## Post #56 by @ian.mcnicoll So I can make this work here 1. Uploaded the template .opt to Better Ehrscape and EhrBase openEHR 2. Committed the XML composition via Better Ehrscape API ``` curl --location --request POST 'https://cdr.code4health.org/rest/v1/composition?templateId=hospitalization_ocean&ehrId=3e674739-950c-4b8a-976b-5aef21c618c5&format=RAW' \ --header 'Content-Type: application/xml' \ --header 'Ehr-Session-disabled: {{Ehr-Session}}' \ --data-raw '