Practical approach to FOLDER OPTs

pablo · 21 October 2022 03:32

Been struggling with this idea for a while, maybe others can help me out.

Since FOLDER is LOCATABLE, it can, and actually should, be archetyped, and since most systems work with templates, there should actually be FOLDER OPTs out there saying how the internal structure of different FOLDERs should look like.

The first challenge with FOLDERs is they are a recursive structure, though is like SECTION or CLUSTER, there is one big difference: for SECTIONs and CLUSTERs the structure is stable once a COMPOSITION is created, and we have an OPT for each type of COMPOSITION.

When we create an EHR.directory instance, it’s a FOLDER, that should have an OPT, though it’s internal structure will keep changing while subfolders are added (at any level of the EHR.directory tree structure) and items are added to the FOLDERs (also at any level).

Creating an OPT for a FOLDER is easy, but then what happens with it’s subfolders at the FOLDER.folders list? Each of those should also be archetyped, but is that archetype mentioned in the top-level/root FOLDER OPT? It could, but then, the folders at the 3rd, 4th and so on, levels, should be also archetype in the same OPT? Or each subfolder could have indeed its own OPT?

It would be easy to say, let’s create a FOLDER archetype that allows anything in its folders attribute, but that won’t help when we want to query the folder structure, since we need concrete archetype ids, constraints and paths to know how to query the structure.

Another option is to have a FOLDER archetype that in the folders attribute has a SLOT to the same archetype ID, creating a recursive archetype dependency (this could also be a CONSTRAINT_REF to the root of the archetype), then use the archetype as-is in the OPT. The issue is if someone needs to add a constraint, then another archetype ID, or version of the current one, should be added to the allowed archetypes in the archetype slot, and this again should allow that new constraint in the first level of the folders, but also in the Nth level, unless a specific constraint doesn’t allow certain archetype IDs, let’s say, at the 3rd level.

What I want to capture as modeling requirements is:

the need of OPTs to formally comply with the openEHR specs
the ability of allowing FOLDER-based queries in AQL and other querying formalisms
enable systems to create FOLDERs at any level of the EHR.directory, for any EHR, and allow to vary the internal structure of EHR.directory from EHR to EHR (different EHRs could have different OPTs for EHR.directory)

I’m just not sure what is a good way of modeling for FOLDERs to allow editing the structure without just having a dummy FOLDER archetype just to be compliant with the specs. I would like to add real constraints to FOLDER archetypes, like the names, but allow users to create folders too.

I guess all this comes from the not-so-explored world of FOLDERs in clinical modeling and querying (we still have open discussions about how AQL over FOLDERs could work and I think it’s because we didn’t use OPTs for FOLDERs extensively).

pablo · 24 October 2022 15:26

@thomas.beale @ian.mcnicoll @Seref @bna @erik.sundvall do you have any opinions about folder modeling?

ian.mcnicoll · 25 October 2022 08:28

I have no personal experience of using FOLDERS but agree that it would be good to be able to define the required structures in a template level

erik.sundvall · 25 October 2022 14:58

Should → Could. I think arcehtyping them could still be optional (a lot of current FOLDERs in EHRs are not archetyped).

Yes, maybe we don’t need to constrain the allowed sub folders (at least not in a first spec iteration). Just being able to add metadata (and query based on it) would be a great addition to Folders and to AQL and cover many use cases.

To clarify, based on the image below:

First maybe we should just try specifying the semantics of archetyping/constraining/querying the “details” property of the FOLDER class. That is likely exactly the same implementation and behaviour as for any other ITEM_STRUCTURE.
Later, in a future version of the specs, maybe specify desired behaviour for archetyping/constraining/querying the “folders” property of the FOLDER class. Implementing that is likely more complicated, especially the querying.

Seref · 25 October 2022 16:59

I think they would complicate things. As usual, you did a great job of identifying various edge cases and all I’m seeing in your summary is complexity. The way I see it, clinicians do not require FOLDERs to express clinical concepts in models, so I’d personally leave it out of scope of archetyping/modelling.

thomas.beale · 25 October 2022 19:51

Not clinical concepts, agree, but standardising ways of arranging / classifying information elsewhere in the EHR, I think you will find that some vendors will want this. DIPS already does it, and I think Code24 does as well; whether they use archetypes or not to create those FOLDER structures I don’t know, but it would be the easiest way to do it i.e. since it can just be done in the same tool as everything else, e.g. Archetype Designer.

The vendors that don’t are probably using some kind of tagging system (e.g. Better).

erik.sundvall · 26 October 2022 05:15

Yes, when building/configuring systems some folder metadata archetyping and associated querying would be very useful for use cases where you want to (possibly hierarchically) group related compositions in a versioned way without changing anything inside the (possibly signed) compositions themselves. Also some metadata is often needed for such groups. I believe @bna has previously detailed how AQL could be extended to use folder based querying with different depths.

In use cases where neither hiearchy nor versioning is needed, I assume that “tags” are enough (if they also can be used to filter queries).

joostholslag · 26 October 2022 06:29

Well, we intend to use folders to model episodes (collections of progress notes). I already had a go at archetyping folders. To add meta-data like, start date, name of episode, etc.etc. I can dit up the folder archetype if helpful.
Main thing missing was to add meaning to a link in the folder. E.g. this problem is the reason for admission.

thomas.beale · 26 October 2022 07:07

This is the ‘how to archetype links’ question. I have some proposals for this.

Seref · 26 October 2022 08:01

Now that deserves a discussion I was going to go into this last night in response to Tom, but decided not to because it would be just a speculative discussion. Not anymore.

I find it interesting that you have a need to model a clinical concept, and you’re looking into FOLDER to express that semantics instead of other types from the RM. Could you comment on why you’re considering the approach you have in mind? (Not implying you’re doing anything wrong, genuinely asking).

These two are related and interesting, I think. FOLDER is fit for purpose to point at something and provide some metadata, but that’s it. By design, it does not employ the approach used in most of the RM: expressing semantics via dedicated types. I know openEHR takes a lot of criticism because of OBSERVATION,EVALUATION etc but they express meaning via dedicated types and that’s a good thing. Both archetyping and AQL are built on this design: define constraints on types to clarify what you mean So they are the meaningful parts in the RM and FOLDER is … meaningless! It’s supposed to point at a meaningful thing, or so is my view.

Given its design, you would probably have to archetype the metadata of the FOLDER, i.e. an ITEM_STRUCTURE to express the meaning you have in mind, and that feels like a square peg in a round hole to me. My impulsive response (not well thought out at all) would be: “would not it be more idiomatic if you had a cluster archetype to which you pointed at with a folder?” At least you’d be archetyping something that is commonly archetyped I.e. a cluster archetype to describe episodic concepts, a folder with N object references, bringing together the compositions and the episode cluster. (though I don’t know what kind of composition would host that episode cluster )

This is all coming from a programmer’s brain, so it may not be valid in the modelling space, but I find myself walking into trouble every time I use different ways of expressing semantics side by side, and the complexity I mentioned in response to @pablo 's question comes from the above situation leading to this mixed-way-of-representing-things. Thanks for allowing me to expand on my answer to Pablo.

Please see the above as just food for thought. It’d be interesting to see how your Folder archetype attempts ended up, if you don’t mind sharing as you suggested. Np if you cannot.

erik.sundvall · 26 October 2022 08:32

Well, using “normal” cluster archetypes for the details* field in FOLDERs would certainly be possible. The enclosing archetype with possible slot(s) for clusters could/would be a FOLDER-based archetype instead of COMPOSITION-based archetype.

That would likely be no different or harder than (what I believe you are proposing) using the items property “pointing” to some specific designated COMPOSITION (among all the ones pointed to) containing a CLUSTER with info about the specific folder.

[*EDIT: see clarifications by @Seref and @ian.mcnicoll below]

Seref · 26 October 2022 08:52

I thought that details is of type ITEM_STRUCTURE. Can we put a CLUSTER archetype there? Unless I’m missing something, that’s not valid as per RM type definitions?

ian.mcnicoll · 26 October 2022 10:18

The way this normally works for Composition/context, is to add a Slot to ITEM_STRUCTURE that can then be used to carry a CLUSTER archetype

Seref · 26 October 2022 10:45

Ok, that’s very interesting . I’ll tread carefully here because I’m sure I am missing something. @thomas.beale can you confirm my understanding of what Ian’s saying: they’re defining a slot for an RM type’s field with a different type than the one declared by RM and there is no subtype relation between ITEM_STRUCTURE and CLUSTER. That would lead to an invalid RM instance being defined/allowed?

I am almost always out of brain cells these days, so some help would be great, how does that work? Maybe I’m misunderstanding the OO/Type level definition of what’s being described here. Or I’m missing some type hierarchy relationship in the below?

Seref · 26 October 2022 11:09

Sigh. To reply to myself, Erik and Ian are talking about using a slot for ITEM_STRUCTURE.items, not for the field with the ITEM_STRUCTURE type. Sorry, I was being too strict in my interpretation of what is meant (I think…)

ian.mcnicoll · 26 October 2022 11:25

It looks as if the tooling is correctly injecting an ITEM_TREE to hold the slot definition,

		context matches {
			EVENT_CONTEXT matches {
				other_context matches {
					ITEM_TREE[at0001] matches {	-- Tree
						items cardinality matches {0..*; unordered} matches {
							allow_archetype CLUSTER[at0002] occurrences matches {0..*} matches {	-- Extension
								include
									archetype_id/value matches {/.*/}
							}
						}
					}
				}
			}
		}
	}

Seref · 26 October 2022 12:02

It looks as if you’re omitting the difference between ITEM_TREE and ITEM_TREE.items

Sorry, I had to

Jokeing aside, many thanks to all for patiently explaining what they mean. It makes sense to use FOLDER this way. The query side is interesting though, where would users of AQL see FOLDER, under an EHR. As a sibling of Compositions?

pablo · 26 October 2022 12:39

Hi all! thanks for your answers and discussions, I see some would like to use FOLDERs in an ad-hoc way, others would like to have just generic archetypes for FOLDERs, and some would like to have very strict structures to use folders to model certain things like care plans or specific types of episodes.

From the querying point of view, not having an archetype would be like not having a schema to query from, so using FOLDERs in an ad-hoc way (not constrained) would lead to also query in an ad-hoc way without knowing the structure of the data being queried, which works if you already now the data, but doesn’t on other use cases.

From the conformance point of view, archetypes are needed, generic or very specific. So we can create test cases for those archetypes and verify:

folders are validated against their archetypes
querying folders based on their archetype definition is possible
if a folder structure is defined in a template, and this is where things get tricky, does the template define the whole EHR.directory structure or not?

For testing conformance too, I would like to define constraints on folder names, folder items, folder subfolders (e.g. can’t contain subfolers, can contain N…M subfolders, 0…, 1…, …), etc.

The challenge I see in OPTs for folders is defining the whole EHR.directory structure in an OPT could be difficult and will result in a rigid structure, I think that is needed but also at some point we need the flexibility of adding a custom/ad-hoc folder somewhere just because a doctor wants to organize their documents or have some organization/classification for researching and teaching, etc.

From the implementation point of view, I’m guessing implementations using folders have other mechanisms than archetypes to define their internal structure and to be able to query that structure.

I’ll try to come up with more concrete examples so we can discuss them.

thomas.beale · 26 October 2022 13:20

Generally speaking, ‘open’ fields have the type ITEM_STRUCTURE, e.g. as you see here (and as you know, but just answering anyway, since others might not realise, and it is a very legitimate question)

We made it the same type in new places, such as FOLDER. It’s easy to deal with in archetyping - see @ian.mcnicoll 's response above - it’s just a wrapping question.

In hindsight, if we had not had ITEM_STRUCTURE and descendant types (clinical types were sure we needed them in 2003!), we would have just (I imagine) connected straight to CLUSTER. We didn’t, and in the intervening years, the clinical modelling community realised they really just wanted trees, which could have just been CLUSTER. So we live with a bit of annoying wrapping going on here - I see it as the price of not having gotten everything right at the start (and maybe not realising early enough that we could potentially ditch the ITEM_STRUCTUREs). It’s life.

In openEHRv2, things would be as you would like (and many others - @erik.sundvall and others have been agitating for this change for years), it’s just that it’s a breaking change from a software point of view, so we have to wait for a new major version of openEHR…

thomas.beale · 26 October 2022 13:21

I see better quality coffee was consumed at some point…!

(Meanwhile I was having my own fail, trying for 45 mins to find VirtualBox guest additions, when I finally realised the .iso was already mounted and just needed to have the executable run… one of those days…)