# What's allowed in FOLDER.items

**Category:** [Specifications](https://discourse.openehr.org/c/specifications/6)
**Created:** 2021-05-07 20:57 UTC
**Views:** 1318
**Replies:** 20
**URL:** https://discourse.openehr.org/t/whats-allowed-in-folder-items/1539

---

## Post #1 by @pablo

Currently FOLDER.items is a List<VERSIONED_OBJECT>, which allows COMPOSITION, EHR_STATUS and FOLDER.

See: https://specifications.openehr.org/releases/RM/Release-1.1.0/common.html#_overview_4

Shouldn't the spec be constrained to List<VERSIONED_COMPOSITION>?

I guess what raises red flags for me is that FOLDER already has a List<FOLDER>, so having VERSIONED_FOLDER instances in FOLDER.items seems strange.

Also the possibility of having EHR_STATUS in the FOLDERS seems strange, unless there is a clear use case for that.

Any comments on this?

---

## Post #2 by @sebastian.iancu

Folders are part of common IM, so besides Compositions might be possible to have also Party items (by refs).
On the other hand It could be a design choice what would be stored there in items, does not feals right to constrain it in the specs. In any case, they are object refs.

---

## Post #3 by @pablo

Folder.items is object_ref but constrained to versioned_object, which could be any of versioned_composition, versioned_folder or versioned_ehr_status. Not sure on what happens at the demographic model.

I'm not following the rest of your comment about the common package.

---

## Post #4 by @thomas.beale

[quote="pablo, post:1, topic:1539"]
Currently FOLDER.items is a List<VERSIONED_OBJECT>, which allows COMPOSITION, EHR_STATUS and FOLDER.
[/quote]

Just to be precise, `FOLDER.items` is of type `List<OBJECT_REF>` (see [here in spec](https://specifications.openehr.org/releases/RM/latest/common.html#_folder_class)).

[quote="pablo, post:1, topic:1539"]
Shouldn’t the spec be constrained to List<VERSIONED_COMPOSITION>?
[/quote]

Normally, it should be references to `VERSIONED_COMPOSITION`, that's true. There's no way to specify that properly in UML unless we introduce more subtypes of `OBJECT_REF`.

[quote="pablo, post:1, topic:1539"]
I guess what raises red flags for me is that FOLDER already has a List, so having VERSIONED_FOLDER instances in FOLDER.items seems strange.
[/quote]

Well, no-one should do this, for sanity ;)

[quote="pablo, post:1, topic:1539"]
Also the possibility of having EHR_STATUS in the FOLDERS seems strange, unless there is a clear use case for that.
[/quote]

Or this...!

---

## Post #5 by @pablo

[quote="thomas.beale, post:4, topic:1539"]
Just to be precise, `FOLDER.items` is of type `List<OBJECT_REF>` (see [here in spec](https://specifications.openehr.org/releases/RM/latest/common.html#_folder_class)).
[/quote]

Yes, I tried to represent what is in the specs: OBJECT_REF with target VERSIONED_OBJECT.

See the UML here https://specifications.openehr.org/releases/RM/Release-1.1.0/common.html#_overview_4

[quote="thomas.beale, post:4, topic:1539"]
Normally, it should be references to `VERSIONED_COMPOSITION`, that’s true. There’s no way to specify that properly in UML unless we introduce more subtypes of `OBJECT_REF`.
[/quote]

The current representation could have target VERSIONED_COMPOSITION instead of VERSIONED_OBJECT.

[quote="thomas.beale, post:4, topic:1539"]
Well, no-one should do this, for sanity :wink:
[/quote]

The issue is what the specs allow, more than what's common sense. I couldn't find something that says "ONLY this is allowed" or "that is NOT allowed".

---

## Post #6 by @sebastian.iancu

Perhaps I'm repeating myself, but I need to express my concerns here.
Those items are not only refs of Versioned_compositions. We (at Code24) took it as was stated in specs for many years already, any kind of OBJECT_REF; in practice we use refs of VERSIONED_COMPOSITIONs as well as VERSIONs. We've also used VERSIONED_PARTY in our Demographics repository, although nowadays we have another alternative solution. My concern is about production data for more than 6y. For us any restriction/constraint added now in the spec will be a breaking change ...and I don't quite see the added value of such change. 
I would see as "unusual" any use of refs other than VERSIONED_COMPOSITION or its VERSIONs, but to forbid it - it seems a step to far.

---

## Post #7 by @pablo

Thanks Sebastian, I'm not really proposing any changes, just trying to check what is valid or invalid, my goal is to  clarify all this so we can create a better conformance test suite.

Your feedback about implementation cases is exactly what we need to figure out the exact rules we might include later, or even state: "this is open to the implementation", which is in fact a specific rule that enables to support anything there.

For instance, if I create a test case where the refs in FOLDER.items can target any LOCATABLE, I will quickly find many of those refs might  not be supported by current implementations, and we need to know if that is valid or invalid in terms of the spec, so we can assure "this is compliant" or "this is not compliant"., since for compliance, even though we can have optional areas, we can't say "don't know if this complies or not".

---

## Post #8 by @thomas.beale

[quote="pablo, post:7, topic:1539"]
Thanks Sebastian, I’m not really proposing any changes, just trying to check what is valid or invalid, my goal is to clarify all this so we can create a better conformance test suite.
[/quote]

Yes, I think this is right - we need to document 'recommended' uses clearly. Code24's original use (the one that creates a kind of double-level versioning) is outside our original intent (as we discussed a lot during previous f2f meetings!), but technically it is just a different interpretation of the same spec, and that we (I...) had not described the intended use of the structures clearly enough.

So I think we need to improve the documentation, and for conformance testing, this would be an instance where you test 'recommended' usages of the model, and don't worry about 'off-label' uses except for some specific product.

---

## Post #9 by @pablo

[quote="thomas.beale, post:8, topic:1539"]
technically it is just a different interpretation of the same spec, and that we (I…) had not described the intended use of the structures clearly enough.
[/quote]

I think we are forgetting about the spec versioning. I mean: Code24 implementation is correct in terms of the version of the spec they picked up to implement. If we correct this in the future, by adding constraints, recommendations, etc., the new version will solve those issues for the future, and the current implementation of Code24 will still be valid against a previous version of the spec. If they want to migrate things to implement the newer version, some refactoring will be needed, but this is will happen to any implementer! For instance, EHRServer is still v1.0.2 only :) and I plan to implement 1.0.4 when some of the infrastructure components are also updated.

---

## Post #10 by @thomas.beale

Actually, the spec of that has not changed for 15y (we added an attribute to FOLDER more recently). Here's 1.0.2.
![release102_folder|556x270](upload://52MngfmDlc6gpQREXGES0cSkfW7.png)

Today:
![image|557x307](upload://z4tIp2d42b4Kc8u6IZXiZ70HO2Y.png)

The idea was always just to maintain *references* to VERSIONED_OBJECTs (representing logical COMPOSITIONs, each with all its versions).

However, the presence of `VERSIONED_FOLDER` probably made it look as if the OBJECT_REFs were meant to point to `VERSIONED_FOLDERs` rather than just `VERSIONED_COMPOSITIONs`. `VERSIONED_FOLDER` is just the type of `EHR.directory`, but not the type of anything inside it.

This [more modern diagram](https://specifications.openehr.org/releases/RM/latest/ehr/diagrams/folders_for_indexing.svg) shows the intention of FOLDERs: each one just points to a collection of VERSIONED_COMPOSITIONs, and also has more FOLDERs beneath.

My memory is (@sebastian.iancu will correct me) that Code24 used the FOLDER.items link to also point to VERSIONED_OBJECTs containing FOLDERs, i.e. introducing versioning of each FOLDER subtree. 

The original design intention what just that the whole directory, i.e. the whole tree, is versioned. Within each version is a FOLDER hierarchy, and those OBJECT_REFs pointing to VERSIONED_COMPOSITIONs.

Code24's interpretation (if I have it right) is not technically incorrect, it just wasn't the expected use of the model - it's a more complicated structure. And we didn't make it that obvious I guess, because there was no detailed explanation in the `directory` package (Common IM)...

---

## Post #11 by @sebastian.iancu

Constraining type of refs would be a restrictive change. Assuming that we will release such change of RM, not only that implementation application logic has to be changed (if something else other then VERSIONED_COMPOSITION was used), but also legacy data will not be compliant anymore with RM specs. Therefore I see this potential change of the specs as a breaking change, part of major release (of course has to be properly analyzed and confirmed in SEC).

---

## Post #12 by @thomas.beale

Agreed - I don't think we should change the technical spec, just improve the documentation.

---

## Post #13 by @sebastian.iancu

[quote="thomas.beale, post:10, topic:1539"]
Code24’s interpretation (if I have it right) is not technically incorrect, it just wasn’t the expected use of the model - it’s a more complicated structure. And we didn’t make it that obvious I guess, because there was no detailed explanation in the `directory` package (Common IM)…
[/quote]

Our creative interpretation was a necessity after working for about 2-3y with EHR.directory to support episodes of care, and care plan in mental care. It proven to be not suitable enough for our use-case, too much overhead data caused be continuously versioning the tree, which sometimes referred to thousands of compositions of one EHR. Our solution was to work with multiple subtrees (versioned) and to bind them virtually in the directory (which acts also as a view) on the application level, whenever the EHR above is opened. It might not be what exactly what the specs textually describes, but it is a valid use case as long as we accept to let the FOLDER.items refs point to VERSIONED_FOLDER also (which was supported by specs). This is the way we used folders for the last 5-6y.

This is now better supported by RM Release-1.1, and we no longer need to bind together subtrees into the main directory, as those subtree are also reachable individually.

However, the `type` of those OBJECT_REF would be good to keep it open. At Code24 we not only refer to latest version of a Composition, but also to a specific version. This can be done by properly setting that attribute. And who knows, there might be future needs to even refer to other types of non-versioned openEHR objects, like GDL/TP - I don't know for sure, just thinking out of the box for now...

---

## Post #14 by @thomas.beale

[quote="sebastian.iancu, post:13, topic:1539"]
too much overhead data caused be continuously versioning the tree
[/quote]

Yes - probably a differential versioning representation would be needed to make this efficient.

[quote="sebastian.iancu, post:13, topic:1539"]
Our solution was to work with multiple subtrees (versioned) and to bind them virtually in the directory (which acts also as a view) on the application level, whenever the EHR above is opened. It might not be what exactly what the specs textually describes, but it is a valid use case as long as we accept to let the FOLDER.items refs point to VERSIONED_FOLDER also (which was supported by specs). This is the way we used folders for the last 5-6y.
[/quote]

Maybe we should just describe this in more detail, with diags in the specs as well?

The only problem though, with multiple uses of the same structure is that they tend to be non-interoperable, e.g. between different vendor systems. But maybe this is one  for which we need different flavours?

---

## Post #15 by @pablo

@thomas.beale @sebastian.iancu did we create any tickets related to this topic?

I just created another thread about the breaking change introduced when the FOLDER.items: OBJECT_REF target was constrained to VERSIONED_OBJECT only in the last RM releases then I found this other thread kind of in the same area.

For instance, the spec should state items can't hold VERSIONED_EHR_STATUS or VERSIONED_FOLDER, but can hold VERSIONED_COMPOSITION and VERSIONED_XXX with XXX demographic.

Thomas mentioned it didn't make sense to reference those types, but the spec allows it, and as long it allows it, any conformance test considering those cases should be considered as correct. An invariant constraint might be useful here.

---

## Post #16 by @thomas.beale

We could try to add invariants that prevent references to things that don't really make sense like VERSIONED_EHR_STATUS, but I am inclined not to. Such references shouldn't break any software - they just won't be useful to the user when they are resolved. And the invariants would not be that easy to write.

I know it's not 100% ideal, but I think we can only constrain so much in the main information models, while managing/minimising implementation complexity.

---

## Post #17 by @pablo

[quote="thomas.beale, post:16, topic:1539"]
Such references shouldn’t break any software - they just won’t be useful to the user when they are resolved.
[/quote]

Shouldn't that be stated somewhere in the specs?

In the conformance area, we need to verify what the spec allows, that means if the spec allows X but it doesn't make any practical sense, we still need to give that as valid when testing conformance. Because if the spec allows it and one implementation didn't realize something doesn't make sense and implemented it as it was specified, then we need to give that implementation a PASS on those tests, we can't say that is wrong if the spec allows it.

If it's not an invariant, it could be a note we can reference from the conformance verification, like the UML allowing X, W, Z, and the note saying but W and Z shouldn't be there. If this is not in the spec, then it is allowed and any implementation will pass with X, W and Z there.

IMO giving all the rules in the spec, without the need of implementers to figure out the rules themselves, is a better way to manage implementation complexity. This whole thread came because I was implementing this and considered the OBJECT_REF could point to anything by the RM 1.0.2 spec. Then in 1.1.0 it seems a rule was added to restrict the target to VERSIONED_OBJECT (which is an undocumented breaking change, see https://discourse.openehr.org/t/detected-a-potential-breaking-change-in-common-im-that-doesnt-seem-to-be-documented/3503 still with that extra constraint, VERSIONED_FOLDER and VERSIONED_EHR_STATUS are valid targets of the OBJECT_REF, so as an implementer I need to add code to actually support those extra types there, with less types I have less code (because in EHRServer and Atomik when you commit a FOLDER with items it checks the targets exists in the DB, and since VERSIONED_X are stored in different tables, I need different queries to check existence, see? more types == more code).

---

## Post #18 by @sebastian.iancu

I still don't understand why we would we need to introduce NOW a restriction (by adding that invariant). What's wrong on keeping it open? The argument that more code is necessary to support such a link, is in my opinion implementation dependent; perhaps there are implementation when supporting VERSIONED_FOLDER besides VERSIONED_COMPOSTION is trivial. 
Maybe if we would had this thoughts 15y ago it would not have been a breaking change - but it is now, a restrictive backwards incompatible change, as invalidates in RM 1.1 data instances that were created in RM 1.0.2.
We can keep this as a best-practice recommendation note (i.e. to not use VERSION_EHR_STATUS, VERSIONED_FOLDER there). In conformance testing nothing particular should be tested (i.e. you don't need to assert and signal error in case of linking those types).

The other aspect I mentioned in 2021 (above) is that OBJECT_REF can point to a VERSIONED_COMPOSITION, but it can also point to a (specific version of a) COMPOSITION (e.g. not the last in the VERSIONED_OBJECT, but one from history) - why is that not a valid use-case (i.e. to bind historical and explicit versions of COMPOSITOINs in a FOLDER)?

---

## Post #19 by @thomas.beale

[quote="pablo, post:17, topic:1539"]
In the conformance area, we need to verify what the spec allows, that means if the spec allows X but it doesn’t make any practical sense, we still need to give that as valid when testing conformance. Because if the spec allows it and one implementation didn’t realize something doesn’t make sense and implemented it as it was specified, then we need to give that implementation a PASS on those tests, we can’t say that is wrong if the spec allows it.
[/quote]

Well we can't test absolutely everything; I'd be inclined to leave out things like a FOLDER pointing a VERSIONED_EHR_STATUS. Technically such a test should always pass, but as you say, it is completely unexpected and meaningless data, so it would be strange to report a conformance failure on such a thing.

We could certainly update the documentation to be more precise - feel free to raise a PR for that. I think we might be able to combine it into an existing CR, since it's pretty minor.

[quote="sebastian.iancu, post:18, topic:1539"]
The argument that more code is necessary to support such a link, is in my opinion implementation dependent; perhaps there are implementation when supporting VERSIONED_FOLDER besides VERSIONED_COMPOSTION is trivial
[/quote]

Right - I think it is likely to be implementation dependent.

[quote="sebastian.iancu, post:18, topic:1539"]
The other aspect I mentioned in 2021 (above) is that OBJECT_REF can point to a VERSIONED_COMPOSITION, but it can also point to a (specific version of a) COMPOSITION (e.g. not the last in the VERSIONED_OBJECT, but one from history) - why is that not a valid use-case (i.e. to bind historical and explicit versions of COMPOSITOINs in a FOLDER)?
[/quote]

Maybe it should be - probably needs to be discussed in SEC.

---

## Post #20 by @pablo

[quote="thomas.beale, post:19, topic:1539"]
Well we can’t test absolutely everything;
[/quote]

Why not? If there is a finite set of possibilities, then it can be tested. Is like not testing all cases in a software because there are too many options, then leave software untested :slight_smile: 

The challenge is to correctly articulate what the spec allows, not the testing part.

[quote="thomas.beale, post:19, topic:1539"]
FOLDER pointing a VERSIONED_EHR_STATUS. Technically such a test should always pass, but as you say, it is completely unexpected and meaningless data, so it would be strange to report a conformance failure on such a thing
[/quote]

The problem is not in the test: if the test expects this to pass and the implementation returns an exception, then it will be considered the system under test has a lower level of conformance with the spec.

You are adding external semantics on the spec interpretation, and that part can't be tested. If this is meaningless data it SHOULD be in the spec.

---

## Post #21 by @pablo

[quote="sebastian.iancu, post:18, topic:1539"]
I still don’t understand why we would we need to introduce NOW a restriction (by adding that invariant).
[/quote]

See https://discourse.openehr.org/t/detected-a-potential-breaking-change-in-common-im-that-doesnt-seem-to-be-documented/3503

That change was introduced recently without much attention on that is a breaking change. So some restriction was already added, before I thought a folder item could point to a VERSION or even a LOCATABLE.

Then the other discussion here is, if the target is only VERSIONED_OBJECT, then can that be ANY VERSIONED_XXX?

Thomas' argument is some VERSIONED_XXX types don't make sense there, which I agree with, though strictly by the spec, they are allowed. In terms of conformance testing, what is not constrained in the spec, is allowed.

[quote="sebastian.iancu, post:18, topic:1539"]
What’s wrong on keeping it open?
[/quote]

I don't see any issues on:

1. allow any target on FOLDER items (including VERSION and LOCATABLEs)
2. include the VERSIONED_OBJECT types that don't make sense there but are still allowed

Though in the case of 2. even if it's not a hard rule to avoid those types as targets, it would be useful to have a comment there in the spec. Then in the conformance spec we can take that comment and say, for conformance that comment is really a hard rule so we don't allow this case to happen (we check it doesn't work or we don't check it at all).

[quote="sebastian.iancu, post:18, topic:1539"]
The argument that more code is necessary to support such a link
[/quote]

Not really the focus of the discussion, my comment wasn't really about supporting links, but that supporting more types on the REF target could lead to more code needed if the software does checks the target exists and different types of targets are stored in different places (e.g. tables).

[quote="sebastian.iancu, post:18, topic:1539"]
a restrictive backwards incompatible change, as invalidates in RM 1.1 data instances that were created in RM 1.0.2.
[/quote]

Agree. That is what I tried to point out in the other thread https://discourse.openehr.org/t/detected-a-potential-breaking-change-in-common-im-that-doesnt-seem-to-be-documented/3503

[quote="sebastian.iancu, post:18, topic:1539"]
The other aspect I mentioned in 2021 (above) is that OBJECT_REF can point to a VERSIONED_COMPOSITION, but it can also point to a (specific version of a) COMPOSITION (e.g. not the last in the VERSIONED_OBJECT, but one from history) - why is that not a valid use-case (i.e. to bind historical and explicit versions of COMPOSITOINs in a FOLDER)?
[/quote]

I think it's valid to have that open and use it as you need, since the object ref could point to a party, locatable and other types (now that I say this, not sure why we have LOCATABLE_REF and PARTY_REF since PARTY is LOCATABLE).

---


**Canonical:** https://discourse.openehr.org/t/whats-allowed-in-folder-items/1539
**Original content:** https://discourse.openehr.org/t/whats-allowed-in-folder-items/1539