Currently FOLDER.items is a List<VERSIONED_OBJECT>, which allows COMPOSITION, EHR_STATUS and FOLDER.
Shouldn’t the spec be constrained to List<VERSIONED_COMPOSITION>?
I guess what raises red flags for me is that FOLDER already has a List, so having VERSIONED_FOLDER instances in FOLDER.items seems strange.
Also the possibility of having EHR_STATUS in the FOLDERS seems strange, unless there is a clear use case for that.
Any comments on this?
Folders are part of common IM, so besides Compositions might be possible to have also Party items (by refs).
On the other hand It could be a design choice what would be stored there in items, does not feals right to constrain it in the specs. In any case, they are object refs.
Folder.items is object_ref but constrained to versioned_object, which could be any of versioned_composition, versioned_folder or versioned_ehr_status. Not sure on what happens at the demographic model.
I’m not following the rest of your comment about the common package.
Just to be precise, FOLDER.items is of type List<OBJECT_REF> (see here in spec).
Normally, it should be references to VERSIONED_COMPOSITION, that’s true. There’s no way to specify that properly in UML unless we introduce more subtypes of OBJECT_REF.
Well, no-one should do this, for sanity 
Or this…!
Yes, I tried to represent what is in the specs: OBJECT_REF with target VERSIONED_OBJECT.
See the UML here Common Information Model
The current representation could have target VERSIONED_COMPOSITION instead of VERSIONED_OBJECT.
The issue is what the specs allow, more than what’s common sense. I couldn’t find something that says “ONLY this is allowed” or “that is NOT allowed”.
Perhaps I’m repeating myself, but I need to express my concerns here.
Those items are not only refs of Versioned_compositions. We (at Code24) took it as was stated in specs for many years already, any kind of OBJECT_REF; in practice we use refs of VERSIONED_COMPOSITIONs as well as VERSIONs. We’ve also used VERSIONED_PARTY in our Demographics repository, although nowadays we have another alternative solution. My concern is about production data for more than 6y. For us any restriction/constraint added now in the spec will be a breaking change …and I don’t quite see the added value of such change.
I would see as “unusual” any use of refs other than VERSIONED_COMPOSITION or its VERSIONs, but to forbid it - it seems a step to far.
Thanks Sebastian, I’m not really proposing any changes, just trying to check what is valid or invalid, my goal is to clarify all this so we can create a better conformance test suite.
Your feedback about implementation cases is exactly what we need to figure out the exact rules we might include later, or even state: “this is open to the implementation”, which is in fact a specific rule that enables to support anything there.
For instance, if I create a test case where the refs in FOLDER.items can target any LOCATABLE, I will quickly find many of those refs might not be supported by current implementations, and we need to know if that is valid or invalid in terms of the spec, so we can assure “this is compliant” or “this is not compliant”., since for compliance, even though we can have optional areas, we can’t say “don’t know if this complies or not”.
Yes, I think this is right - we need to document ‘recommended’ uses clearly. Code24’s original use (the one that creates a kind of double-level versioning) is outside our original intent (as we discussed a lot during previous f2f meetings!), but technically it is just a different interpretation of the same spec, and that we (I…) had not described the intended use of the structures clearly enough.
So I think we need to improve the documentation, and for conformance testing, this would be an instance where you test ‘recommended’ usages of the model, and don’t worry about ‘off-label’ uses except for some specific product.
I think we are forgetting about the spec versioning. I mean: Code24 implementation is correct in terms of the version of the spec they picked up to implement. If we correct this in the future, by adding constraints, recommendations, etc., the new version will solve those issues for the future, and the current implementation of Code24 will still be valid against a previous version of the spec. If they want to migrate things to implement the newer version, some refactoring will be needed, but this is will happen to any implementer! For instance, EHRServer is still v1.0.2 only 
and I plan to implement 1.0.4 when some of the infrastructure components are also updated.
Actually, the spec of that has not changed for 15y (we added an attribute to FOLDER more recently). Here’s 1.0.2.
Today:
The idea was always just to maintain references to VERSIONED_OBJECTs (representing logical COMPOSITIONs, each with all its versions).
However, the presence of VERSIONED_FOLDER probably made it look as if the OBJECT_REFs were meant to point to VERSIONED_FOLDERs rather than just VERSIONED_COMPOSITIONs. VERSIONED_FOLDER is just the type of EHR.directory, but not the type of anything inside it.
This more modern diagram shows the intention of FOLDERs: each one just points to a collection of VERSIONED_COMPOSITIONs, and also has more FOLDERs beneath.
My memory is (@sebastian.iancu will correct me) that Code24 used the FOLDER.items link to also point to VERSIONED_OBJECTs containing FOLDERs, i.e. introducing versioning of each FOLDER subtree.
The original design intention what just that the whole directory, i.e. the whole tree, is versioned. Within each version is a FOLDER hierarchy, and those OBJECT_REFs pointing to VERSIONED_COMPOSITIONs.
Code24’s interpretation (if I have it right) is not technically incorrect, it just wasn’t the expected use of the model - it’s a more complicated structure. And we didn’t make it that obvious I guess, because there was no detailed explanation in the directory package (Common IM)…
Constraining type of refs would be a restrictive change. Assuming that we will release such change of RM, not only that implementation application logic has to be changed (if something else other then VERSIONED_COMPOSITION was used), but also legacy data will not be compliant anymore with RM specs. Therefore I see this potential change of the specs as a breaking change, part of major release (of course has to be properly analyzed and confirmed in SEC).
Agreed - I don’t think we should change the technical spec, just improve the documentation.
Our creative interpretation was a necessity after working for about 2-3y with EHR.directory to support episodes of care, and care plan in mental care. It proven to be not suitable enough for our use-case, too much overhead data caused be continuously versioning the tree, which sometimes referred to thousands of compositions of one EHR. Our solution was to work with multiple subtrees (versioned) and to bind them virtually in the directory (which acts also as a view) on the application level, whenever the EHR above is opened. It might not be what exactly what the specs textually describes, but it is a valid use case as long as we accept to let the FOLDER.items refs point to VERSIONED_FOLDER also (which was supported by specs). This is the way we used folders for the last 5-6y.
This is now better supported by RM Release-1.1, and we no longer need to bind together subtrees into the main directory, as those subtree are also reachable individually.
However, the type of those OBJECT_REF would be good to keep it open. At Code24 we not only refer to latest version of a Composition, but also to a specific version. This can be done by properly setting that attribute. And who knows, there might be future needs to even refer to other types of non-versioned openEHR objects, like GDL/TP - I don’t know for sure, just thinking out of the box for now…
Yes - probably a differential versioning representation would be needed to make this efficient.
Maybe we should just describe this in more detail, with diags in the specs as well?
The only problem though, with multiple uses of the same structure is that they tend to be non-interoperable, e.g. between different vendor systems. But maybe this is one for which we need different flavours?
@thomas.beale @sebastian.iancu did we create any tickets related to this topic?
I just created another thread about the breaking change introduced when the FOLDER.items: OBJECT_REF target was constrained to VERSIONED_OBJECT only in the last RM releases then I found this other thread kind of in the same area.
For instance, the spec should state items can’t hold VERSIONED_EHR_STATUS or VERSIONED_FOLDER, but can hold VERSIONED_COMPOSITION and VERSIONED_XXX with XXX demographic.
Thomas mentioned it didn’t make sense to reference those types, but the spec allows it, and as long it allows it, any conformance test considering those cases should be considered as correct. An invariant constraint might be useful here.
We could try to add invariants that prevent references to things that don’t really make sense like VERSIONED_EHR_STATUS, but I am inclined not to. Such references shouldn’t break any software - they just won’t be useful to the user when they are resolved. And the invariants would not be that easy to write.
I know it’s not 100% ideal, but I think we can only constrain so much in the main information models, while managing/minimising implementation complexity.
Shouldn’t that be stated somewhere in the specs?
In the conformance area, we need to verify what the spec allows, that means if the spec allows X but it doesn’t make any practical sense, we still need to give that as valid when testing conformance. Because if the spec allows it and one implementation didn’t realize something doesn’t make sense and implemented it as it was specified, then we need to give that implementation a PASS on those tests, we can’t say that is wrong if the spec allows it.
If it’s not an invariant, it could be a note we can reference from the conformance verification, like the UML allowing X, W, Z, and the note saying but W and Z shouldn’t be there. If this is not in the spec, then it is allowed and any implementation will pass with X, W and Z there.
IMO giving all the rules in the spec, without the need of implementers to figure out the rules themselves, is a better way to manage implementation complexity. This whole thread came because I was implementing this and considered the OBJECT_REF could point to anything by the RM 1.0.2 spec. Then in 1.1.0 it seems a rule was added to restrict the target to VERSIONED_OBJECT (which is an undocumented breaking change, see Detected a potential breaking change in Common IM that doesn't seem to be documented still with that extra constraint, VERSIONED_FOLDER and VERSIONED_EHR_STATUS are valid targets of the OBJECT_REF, so as an implementer I need to add code to actually support those extra types there, with less types I have less code (because in EHRServer and Atomik when you commit a FOLDER with items it checks the targets exists in the DB, and since VERSIONED_X are stored in different tables, I need different queries to check existence, see? more types == more code).
I still don’t understand why we would we need to introduce NOW a restriction (by adding that invariant). What’s wrong on keeping it open? The argument that more code is necessary to support such a link, is in my opinion implementation dependent; perhaps there are implementation when supporting VERSIONED_FOLDER besides VERSIONED_COMPOSTION is trivial.
Maybe if we would had this thoughts 15y ago it would not have been a breaking change - but it is now, a restrictive backwards incompatible change, as invalidates in RM 1.1 data instances that were created in RM 1.0.2.
We can keep this as a best-practice recommendation note (i.e. to not use VERSION_EHR_STATUS, VERSIONED_FOLDER there). In conformance testing nothing particular should be tested (i.e. you don’t need to assert and signal error in case of linking those types).
The other aspect I mentioned in 2021 (above) is that OBJECT_REF can point to a VERSIONED_COMPOSITION, but it can also point to a (specific version of a) COMPOSITION (e.g. not the last in the VERSIONED_OBJECT, but one from history) - why is that not a valid use-case (i.e. to bind historical and explicit versions of COMPOSITOINs in a FOLDER)?
Well we can’t test absolutely everything; I’d be inclined to leave out things like a FOLDER pointing a VERSIONED_EHR_STATUS. Technically such a test should always pass, but as you say, it is completely unexpected and meaningless data, so it would be strange to report a conformance failure on such a thing.
We could certainly update the documentation to be more precise - feel free to raise a PR for that. I think we might be able to combine it into an existing CR, since it’s pretty minor.
Right - I think it is likely to be implementation dependent.
Maybe it should be - probably needs to be discussed in SEC.
Why not? If there is a finite set of possibilities, then it can be tested. Is like not testing all cases in a software because there are too many options, then leave software untested
The challenge is to correctly articulate what the spec allows, not the testing part.
The problem is not in the test: if the test expects this to pass and the implementation returns an exception, then it will be considered the system under test has a lower level of conformance with the spec.
You are adding external semantics on the spec interpretation, and that part can’t be tested. If this is meaningless data it SHOULD be in the spec.
