For the moment I can think on possible problems when using that kind
of id if someone tries to put the id as the file name in some OS. ':'
is not a valid character in a file name.
Hi Diego,
That’s an issue which has been raised in previous discussions, and was resolved by allowing other characters to be used in filenames e.g. ∼
- see http://www.openehr.org/svn/specification/TRUNK/publishing/architecture/am/knowledge_id_system.pdf - right at the end of the document
There is perhaps an argument for using ‘∼’ for both the internal identifier and the filename but I don’t think we should insist that the filename must replicate the form of the extendedID precisely. In some cases on local systems might want to omit revision number or even the namespace, depending on circumstances. The source of truth for any tooling or application must be the archetype metadata, not the filename.
Ian
Indeed, but I'm sure that it will cause more than a headache to some
newcomers 
Diego, we commented about archetype naming and archetype id in the lists not so long ago, one thing was clear: adl file names and archetype ids are independent, but is useful to have the id on the file name when developing a system. This is not so useful for production systems (the ones I believe will use the full ID mentioned by Ian).
Diego, it is a Windows limitation. My OS has no problem with ':' in a filename.
But besides from that.
I don't think we should connect the place where or the way how the archetype is stored with the way to identify the archetype.
For you as a archetype-editor-designer, it can be convenient to have the ID as filename, I think you will find a solution for that.
But in my opinion, we should not let that be a part of this discussion.
Bert
Hi Ian,
For me the big advantage of internally verifiable archetypes is that you don't need any organization to be sure that the archetype you are looking at is the same archetype with which you defined a stored dataset.
I think it is important to let be possible that archetypes are used with no connection to the outer world, and that they are still are verifiable.
You don't want a hospital to remain out of reliable data because the Internet is down?
Because that is the case if an archetype cannot proof itself to be reliable.
That is why I think having MD5 keys in the metadata is a good idea.
Also, imaging archetypes being used in very special medical organizations which have nothing to do with the outer world. For example, archetypes needed in industry or specialized organizations.
The advantage, in my view, of multi level modeling, this is possible without having to write expensive systems. Just write an archetype and you are in business. We should not lose focus on this flexibility.
But I also see your problem of development, testing, data becoming useless because MD5 keys are changed because of archetypes revisions.
is fixed, it may never change again.
There is something to say for that.
Maybe it is just right that you cannot use a revised version to retrieve/interpreted older data. You should use the original archetype, identified by the MD5 key.
I think this is a strong point.
Bert
(and if you cannot find it anymore, and it is impossible to write one with the same key, there should be an escape-API, like there should also be for development and testing situations)
I think the clinical modelling community on CKM have worked out a good life cycle now. That has taken some years to evolve.
I doubt if they would claim it is the final answer for the whole world, nor even all uses of archetypes.
So I would say a standard for this is still premature, but that if anyone wanted to create a draft of such a standard, it would be hard to imagine one of the candidate life cycles not being the one developed in CKM, with its hundreds of users.
- thomas
that is more or less what we do in our own implementations, and it has taken some time to work out even how to make this work properly. For example, doing an MD5 of the whole archetype actually doesn’t work. You have to first generate a canonical version containing only the ‘stuff that matters’. Then MD5s can be useful. But we also need a proper 3-part version identification system. MD5s don’t replace this, they just tell you that the copy you have of X is really X (assuming X’s MD5 is published in a place you trust), or else a guaranteed equivalent of X (e.g. in Dutch). I.e. the usual integrity check. And/or non-repudiation, if you including signing. But getting the definition of the canonical form is not that easy. Anyway, there are two useful docs that I and Ian McNicoll will get posted ASAP:
I know, we changed that constraint some time ago, I'm just saying it's
something that could happen if people is not aware.
Hi Bert,
I can understand Tim’s point from an implementation perspective, and it really all depends on what you mean by ‘an archetype’ - is this a Version of an archetype, a revision of an archetype or a specific build of an archetype. I have seen enough real implementation of openEHR to be confident that the version/revision/build rules work very well, in terms of clearly defining breaking and non-breaking change. One of the joys of openEHR development is slipstreaming in a revision archetype into a running system and watching it carry on running safely but able to support the expanded dataset afforded by the new revised archetype, without having to change any database schema or legacy queries. On the other hand, I can also understand that others might want to take a different view in certain mission critical areas of specifying a specific revision or even build of a particular archetype for data collection and querying. Any proposals have to support both perspectives.
The problem with the strict MD5 approach is that you are going to have to update all of your software and query references every time you change your schema and effectively use a ‘new archetype’, in many respects this is even worse than the RDBMS approach which at least generally allows new columns without breaking queries etc.
So, I think I probably disagree with Tim, except in the sense that I think we can have the best of both worlds if we adopt a sufficiently flexible id policy that lets implementers specify the exact degree of control they want to apply.
Regards,
Ian
Thomas, it is also my idea that a MD5 on an whole archetype does not work. I wrote it a few times before on this list. You must have missed it, no problem, I also don’t read everything.
Most important is an MD5 over the definition, it must be done after removing comments and trailing spaces, line-ends, etc.
Other MD5 could be taken on the ontology, one for every language.
Then the second part of your message, how needs an ID to be defined?
I do not have a strong opinion on that. Where should information be, in the ID or in the metadata?
I think if an ID contains obvious information, the chance that it will be unique, will be very small.
I think you better call it “name”, because it describes an archetype, it does not identify an archetype.
I think that this kind of semantics is important, but it is not the most important part of the discussion, the most important part should be which information to include and use and for which purpose, and then, as detail, where that information should be, and how to call it.
Bert
sorry, I missed a few things in the avalanche One theory I have is to convert the archetype to a standard dADL serialisation (the ADL Workbench already does this), and throw out descriptive elements, but I think the ontology has to be included. I’ll get the id proposal up and let’s discuss it then… - thomas
Ian,
Joy is a good thing, but once an archetype is used in production, one must be able to find that archetype back, and proof that this is the same.
Not a revision, not a version, but that specific archetype.
I guess every informatic-specialist will agree on this.
But this is an easy requirement, check the MD5.
Having an revision does not need to mean that the pre-revision does not anymore exist.
It is something that should coexist with the new revision, which can be recognized as an new revision and have the precedence on new development, but the old version should remain available.
Things will really get complicated if we have to check what the difference is between the archetype of a stored dataset, and the X-th revision of that archetype.
I don’t think we should want these complications.
I do agree that changing archetypes while using them in an application can cause problems. But also, using different versions of an archetype inside one application is not that complicated. Just regard them as different archetypes, and you are half way. The strict MD5 will help you doing so. Mistakes are not possible. That is good news. Strict MD5 checking will help the application-builder.
Best regards
Bert
OK
Bert
Hi Bert,
Comments inline.
Joy is a good thing, but once an archetype is used in production, one must
be able to find that archetype back, and proof that this is the same.
Not a revision, not a version, but that specific archetype.I guess every informatic-specialist will agree on this.
But this is an easy requirement, check the MD5.
IAN : Agreed, and that is exactly how we use MD5 hashing in Ocean
applications
Having an revision does not need to mean that the pre-revision does not
anymore exist.
It is something that should coexist with the new revision, which can be
recognized as an new revision and have the precedence on new development,
but the old version should remain available.
IAN: Also agreed
Things will really get complicated if we have to check what the difference
is between the archetype of a stored dataset, and the X-th revision of that
archetype.
I don't think we should want these complications.
IAN: The whole point of the Version/ Revision rules is that this check
takes place at design-time, and is not needed at run-time. Currently
within CKM and other authoring tools, we do exactly these kind of
comparison checks every time we upload a new archetype Build, and this is
what allows us to determine whether a new Version is required ( effectively
a new archetype) or whether the changes introduced will be compatible with
data stored with the previous revision. Of course, you have to realy on the
archetype authors to be adhering to these rules correctly and applying the
correct semantic versioning identifiers. We, and others, have any number of
systems and applications which work quite succesfully and safely on this
basis.
I do agree that changing archetypes while using them in an application can
cause problems. But also, using different versions of an archetype inside
one application is not that complicated. Just regard them as different
archetypes, and you are half way. The strict MD5 will help you doing so.
Mistakes are not possible. That is good news. Strict MD5 checking will help
the application-builder.
IAN: If you take the approach of regarding every non-breaking revision as a
new archetype, you are effectively negating one of the huge advantages of
openEHR, in that the formalism and versioning rules allow for considerable
modifications to be made without breaking existing code and queries.
So I can extend an archetype over several revisions, continually adding new
content, but this query for All of a patient's diastolic blood pressures
will remain completely valid (pseudo AQL-for clarity)
SELECT diastolic FROM EHR CONTAINS openehr.org:
:OPENEHR-EHR-OBSERVATION.blood_pressure.v1
if we are going to query on the basis of identifying the archetype via a
strict MD5Hash, the query (or any other reference to this concept will have
to be updated to cater for every new archetype, worse still every single
minor typo change or language addition, that is of no interest
e.g.
SELECT diastolic FROM EHR CONTAINS "7CEDD2FF334E6DE44B16A369F14AC800" OR
"CEDD2FF334E6DE44B16A369F1EE123" OR "ABCD2FF334E6DE44B16A369F1EE143"
This is exactly the kind of overhead that traditional RDBMS approaches
introduces and which makes openEHR much more agile in being able to respond
to changing clinical requirements without requiring change to existing
software.
We know this works, but it does require careful control of operational
repositories, and we all agree that the current metadata is inadequate, and
needs to be improved to prevent the kind of naming and versioning
collisions you are rightly concerned about. Part of the delay in getting
this specified is that any mechanism has to cope with both the requirements
of the archetype and template development cycle, and of course,
implementation, which has taken some time to understand clearly.
I expect MD5 Hashing to be part of the solution, but it cannot be the only
solution. I cannot see any reason why the kind of close control you want
cannot be acheived, whilst allowing the greater flexibility which is need
for development cyclel and to support agile implementation
Regards,
Ian
Hi Ian,
Comments inline.
Very good, what I propose is to facilitate this proces by adding MD5 to the archetype, so that the archetype itself can proof it is the one you want. It is a safety-measure.
With you, I am in the opinion that archetypes should be revised if necessary. I am just saying that you should not use a revised archetype to interpreted data which are stored previous to that revision. There is no need to do that, it is only a risk.
No, I do not agree with that. You should always use the best archetype to store data. And if the revised is better (which mostly is the case), you should use that. I do not want to stand in the way of progress.
Yes, that is true, repairing typo’s will be more difficult. But if you do not check the MD5, then you have to check and judge what the reason is for revision.
There is no solution without disadvantages. And again, there is no reason for not using the old archetype for interpreting data which are stored with that old archetype.
Maybe, if the typo is that much disturbing that you don’t want to use the old archetypes anymore, you can also store again the data with the new archetype.
And besides that, typo’s are in the ontology section, not in the definition (that is syntaxly checked), because that, I suggested separated MD5 checks, for the ontology section, for each language, and one for the definition.
So, if there is found a typo and the MD5 for the ontology for a specific language does not fit, but the archetype indicates by other metadata that it is a revision, than it can safely be used, also for data stored with the unrevised version.
But again, there is no need to do that, because using the original archetype is always safe. We must accept that history has things in it, which are considered wrong later. The storage of data in that time was in the context of that archetype at that time.
A good idea would be the mention the revision change in the metadata, in a special section.
Bert