Hi,
I must have been overlooking it all the time, but I cannot find the formal definition of the OET-XML
Thanks for helping me out.
Bert
Hi Bert,
this page <http://htmlpreview.github.io/?https://github.com/openEHR/specifications/Release-1.0.2/publishing/its/XML-schema/index.html>should help it's the 'XML schema' link from the specifications page <http://www.openehr.org/programs/specification/releases/1.0.2>, near the bottom.
- thomas
Hi Thomas, thank you very much. I found this link already: But I don’t know how to understand this schema in context with OET-files I found on CKM. In those OET-files are elements like definition, integrity-checks, rules, item, items, and much more. It seems not to fit on the contents of this XSD which does not define those elements. Do I misunderstand something? Bert
Sorry for my bad english ;-(
I have, in context of my previous question some more questions.
Is it the case that there are no formal definitions for OET and for OPT?
Is it maybe because both are not yet definitely defined? That is a good reason.
I would like to see the formal XSD-definition the Template-designer works with and which results are published on CKM and used in several community-projects.
Is that possible? I will take in account that they are not the definite version.
Another question:
I wonder why there are two artefacts, OET and OPT. Both contain different information, but also have a part of overlap/redundancy.
Isn't it possible to combine the two to one formal definition for an archetype?
Thanks
Bert
to one formal definition for an template?
Excuse me for the error ;-\
Hi Bert,
My understanding only - but I'll give it a go to answer your questions as good as I can.
I have, in context of my previous question some more questions.
Is it the case that there are no formal definitions for OET and for OPT?
Is it maybe because both are not yet definitely defined? That is a good reason.
OPT just uses the reference model xsds I believe: https://github.com/openEHR/reference-models/tree/master/models/openEHR/Release-1.0.2/XSD
OET: There is a OET parser in the Java Implementation, see here: https://github.com/openEHR/java-libs/tree/master/oet-parser
It also includes the xsds: https://github.com/openEHR/java-libs/tree/master/oet-parser/src/main/xsd - CompositionTemplate.xsd is the main one. The parser is then just generated
That said, OETs (or their equivalent) will be expressed pretty similar to how specialised archetypes are expressed in ADL 2.0 source format I believe - see here for a starting point: http://www.openehr.org/wiki/display/ADL/ADL+1.5+templates+as+single+artefacts
I would like to see the formal XSD-definition the Template-designer works with and which results are published on CKM and used in several community-projects.
Is that possible? I will take in account that they are not the definite version.
You can generate the TDS which I think you are referring to from CKM directly for any COMPOSITION template.
Go to the Export Template tab for a Composition template, e.g. http://openehr.org/ckm/#showTemplate_1013.26.2_EXPORT and download it from there. It is generated for you on the fly.
Another question:
I wonder why there are two artefacts, OET and OPT. Both contain different information, but also have a part of overlap/redundancy.
Isn't it possible to combine the two to one formal definition for an archetype?
OPT would be generated from OET+the archetypes. So essentially it is combining archetypes and restricting them further. Not much different to a specialisation of an archetype. So, compare OET as just listing the differences and OPT as the full specification. OPT is just generated from the oet + the underlying archetypes as required. During deployment you are likely only interested in the OPT, but during the development of archetypes & templates, you are much happier dealing with a file format that just restricts the archetypes for your use case, but which doesn't carry the whole information from the archetypes - that would get you into a maintenance nightmare.
Hope this helps and clarifies
Cheers
Sebastian
Hi Bert,
You are correct - this is not the schema for .oet but an old very first draft of a template schema which to my knowledge has never been used and should probably be withdrawn.
The schema for the operational template 1.4 .opt format is available and I think that should be uploaded in its place.
I am not sure that a schema for the .oet format exists.
.opt is the key format
Ian
Thanks Sebastian,
I had not appreciated that the various flavours or Template are published in the Java implementation. I suggest that we use this as the source of truth.
https://github.com/openEHR/java-libs/tree/master/oet-parser/src/main/xsd
CompositionTemplate.xsd - .oet format
https://github.com/openEHR/java-libs/blob/master/oet-parser/src/main/xsd/CompositionTemplate.xsd
This is the native format that Template Designer uses internally . It is a diff i.e. it carries only the constraints on the underlying archetypes, not any of the archetype info itself. This isa only ever used as a design artefact.
Template.xsd - .opt format
https://github.com/openEHR/java-libs/blob/master/oet-parser/src/main/xsd/Template.xsd
This is the technical artefact that most Template-supporting systems e.g OceanEHR, Marand Think!EHR, Cabolabs, Code 24 etc use. . It is is essentially a wrapper round the existing AOM schema, to support aggregation and further constrinty of the archetypes. It is essentially a 'compiled and flattened form which fuses the diff constraints of the .oet with the underlying archetypes. In that sense it is just a giant archetype, which internally is 90% compatible with the AOM (and imports the same AOM schema).
In practice this means that if I create a .opt , I can upload this to one of these systems and it will be ready to use, with no need to upload or manage the individual archetypes.
.opt is also used to generate other downstream artefacts such as GUI frameworks and code-stub libraries. The other schema that is generated is Template DataSchema
In ADL 2.0 terms, the .oet will be replaced by differential ADL, but the .opt will remain as a final ‘compiled’ version of the template for technical use.
As part of the tooling re-development we should set up a wiki page to document the various favours of template and template-related schema.
Ian
Ian
Thanks Sebastian and Ian,
Ian, you are right, the XSD Thomas linked to is confusing and should have been withdrawn.
Indeed, for the OPT you can go to the AOM-XSD.
In fact, an OPT is like an XML-archetype, nothing more. It has the option to include more archetype-Id's, in fact, a lot like slots but then with no wildcards.
But it is not an archetype, it is a template.
When you look at templates this way, it looks a lot like a slotted archetype, but in fact, the paths which can be deduced are in fact paths to external archetype's.
That is not very convenient for programmer's. They have to walk through the chained elements to calculate/discover the paths where the data are.
For that, the OET format is much more convenient.
You can generate the TDS which I think you are referring to from CKM directly for any COMPOSITION template
I looked at it, and I discovered it already before. But it is not what I am looking for. It creates an XSD for that specific template, where datafiles for that template should comply to.
I am looking for an XSD which is a formal definition of the OET format itself.
If I want to write OET-using software, I must be able to validate if a template to be used is according the rules. And that I cannot find.
CompositionTemplate.xsd is the main one
I looked at the CompositionTemplate.xsd, it has a lot of elements the OET-files have, but I can see, since then, a lot has changed. But it gives some anchors for understanding. 
OPT would be generated from OET+the archetypes. So essentially it is combining archetypes and restricting them further
but which doesn't carry the whole information from the archetypes - that would get you into a maintenance nightmare.
I don't see that point. The information/constraints the archetype has, can be taken care of at design time, and easily checked afterwards for validity.
For example, cardinality and occurrences/existence are easy to express in OET, without much extension in the definition of the OET-XSD.
I see a lot of space used in the elements is used by the occurrences-definition.
XML-Schema has the exact same mechanism as attributes which is less verbose.
Cardinality should also use this mechanism, and then extended with two extra-attributes "ordered" and "unique"
Same for archetype_id and node_id.
Of course XML-parsers are very much optimized for validating on normal XSD-mechanism. Reading attributes cost less memory and less processor time then reading element-structures. And the parsed document is much smaller, so the used memory is smaller, and this allows for DOM-parsing instead of SAX-parsing. DOM-parsing has much more flexibility and speed, but it needs to take the whole document in memory
I think the OPT is too faithfully a translation of the AOM, and that is why it becomes a nightmare. It is not optimized at all.
And then we have the constraints on primitives. I think there are four kinds, list, interval, pattern and exact value. Maybe I forgot one. I don't see how these small items can change the OET-format into a nightmare.
So this all would already clean up quit a bit of nightmare, without losing any information or expression.
It can be that I overlook a lot things, and if, I am very eager to see what.
I have the problem of a missing formal definition of the template-model, and now I learn that there is a risc of a nightmare, when I combine information which is handled separated.
I wonder if that is true.
I am now in a difficult situation for me, and I have to take some longterm development decisions regarding template-functionality, very soon.
Should I follow and wait for the OpenEHR developments, or should I go my way?
Thanks
Bert
Hi Ian,
I had not appreciated that the various flavours or Template are published in the Java implementation. I suggest that we use this as the source of truth.
https://github.com/openEHR/java-libs/tree/master/oet-parser/src/main/xsd
I am sorry to say that this is not possible, they are also outdated, but less then the link I got this morning.
But "source of truth" sounds formal. I think the suggestion is very suddenly for such a formal decision.
emplate.xsd - .opt format
https://github.com/openEHR/java-libs/blob/master/oet-parser/src/main/xsd/Template.xsd
The match between practice and definition is quite good, but also needs some revision, I validated some OPT-files with the template.xsd by using Saxon EE.
It came to reoccurring errors, which are only a few errors, but reoccurring and then it become many.
And, to take in regard, XML-Schema is not the easiest way of defining a formal standard, you see some unnecessary restrictions.
For example, many errors come from the use of "xs:sequence" in the definitions, while "xs:choice" is what is wanted.
"xs:sequence" forces a defined sequence in an instance, and that is not what is wanted, as we can see on the results on CKM.
Thereby, I think the OPT-definition is not good enough for production-use. I explained in my message to Sebastian.
It has a lot of unnecessary overhead, and when dealing in a busy environment you should maybe want to reduce that.
This illustrates some moving targets.
This is the technical artefact that most Template-supporting systems e.g OceanEHR, Marand Think!EHR, Cabolabs, Code 24 etc use.
I don't know how these organizations handle the missing formal definitions.
Do they wait for formal definitions and in the meantime handle which seems wise to their programmers?
They don't tell us of course, it is commercial interest to follow formal definitions which do not change.
But also it is important that they need to deliver stable software now, and there is some friction between the two.
And they need to make long term decisions on which customers can rely.
In practice this means that if I create a .opt , I can upload this to one of these systems and it will be ready to use, with no need to upload or manage the individual archetypes.
Ah, now I understand, an OPT is in fact an archetype. The system does not need to know the underlying archetypes, is your argument, because the template has all of the information.
But still I don't see why the OET & OPT artefacts exist. The OPT, designed better, more efficient, and convenient for programmers would do on its own.
I also see another problem. How can a system be sure if a template complies with an archetype if it does not have that archetype?
But Thanks a lot, Ian and Sebastian,
You explained a lot, and helped me making some decisions.
Bert
This is in fact an interesting thought. It could mean the end of archetypes. They only need to exist virtual, as inner-parts of templates.
The system does not need to know them, as long as the template designer says they exist, somewhere, and on other existence, they are the same.
It will be the templates which take over control, the decide what is right, they decide the ADL-paths, they decide the structure, and there is no way to validate if they are right.
And the naming of archetypes in file-names.
As archetypes do not need to be managed anymore, then they don't need to be in files. We had that discussion just a few weeks ago.
Virtual Archetypes!
Suppose you are wrong, archetypes are still needed on a system, suppose you fantasized this purpose, then the existence of the OPT-templates would be useless.
Why need OPT-templates if all the information already is on a system.
I therefor think you are right, I think this is really the purpose of the OPT-templates. I cannot think of another purpose.
Quite some conclusion is that.
except for one tiny detail - queries are built using archetype ids and paths, not templates. The data need to know the archetype ids, so that querying works. This is fundamental - it is what allows (say) BP data created using the BP archetype in 100 different templates to still be queried . In other words, no matter what template you build tomorrow containing a given archetype, all queries for the data in that archetype will automatically work with the newly created data of that template. OPTs are a compiled form of a template, including all of its archetype information, in an efficient form. In the near future, we will define different ways of generating OPTs that do things like:
This is in fact an interesting thought. It could mean the end of archetypes. They only need to exist virtual, as inner-parts of templates.
The system does not need to know them, as long as the template designer says they exist, somewhere, and on other existence, they are the same.
Exactly.
However, you are also correct that you run the risk of having different templates using different revisions and even versions of archetypes. At the moment that is mostly managed by clearly separating the opt creation process from day-to-day archetype and templating design. This is not really any different from managing the archetypes as individual entities. There does have to be careful control of the versioning. The new numbering rules should make it easier for this to be well-managed, and to warn of potential conflicts. The templates will equally have to have careful versioning rules, though I would expect this just to bubble-up from what is happening to child archetypes i.e if any of the children has a major revision, the template must have a major revision. This is not something that CKM does right now but it should be possible.
You are correct, you can get to the same place by managing all of the component archetypes as a set, but since many will be diffs, you are going to have to compile them to a flattened form at some point - .opt is just an XML representation of that flattened form.
It is really the choice between building a piece of OS software from source or downloading the installer 
Archetype and template consumers may not need the source files but authors like me definitely do. When you agree to give up managing your source code as files , I will do the same Seriously there are huge advantages in keeping things as independent files for authors but I agree no need for these to be managed onside operational repositories.
The idea of ADL 2.0 is to replace .oet with pure ADL - there is flattened for of templated ADL and ongoing work to be able to produce .opt from that.
There is some thought about changing the .opt form to be less wordy, and your input would be very helpful if you wanted to get involved.
Ian
Thanks for this explanation. Good idea, you are right. Concluding: nothing really changed, just a compiled template definition. No big deal, in fact.
I happen to have read the XSD-bible: "Definitive XML Schema 1.1" by Priscilla Walmsley
I worked a lot with it last couple of years. So I can certainly give some advise, when time comes.
Just let me know.
Bert
Thanks Bert,
The time has come …
Sign up at http://www.openehr.org/wiki/display/dev/Online+archetype+and+template+tools
or email Erik Sundvall, who is coordinating this effort ..
Thomas is, of course correct, in that the use of .opt does not change the way that querying works or the semantics but it is hugely helpful in simplifying delivery of complex content definitions into production systems.
Ian
Ian
I checked it, never heard of Adobe Connect, but it seems to work.
So I come and see what I can do next meeting, I believe it is at noon Swedish time.
Bert