Consequences of bad modelling and lessons learned?

Clinical
, ,
1

Hi everyone,

I would like to have a discussion about the consequences of bad modelling.
Usually if the modelling is good we know how beautiful that can be, but what happens if it’s not? What are the further issues? What are the lessons learn? What was done to avoid these situations?
For me openEHR it’s really great and has an awesome proposition to address many issues on current healthcare IT situation - which it’s why I decided to go further on working with this technology, although it’s not a easy one to start and requires some special knowledge and time to learn. But nevertheless, I think it is really necessary to have an open topic about what can happen when bad modelling is being done and the further consequences on AQL’s and deployment on different instances of openehr platforms.
(I will also try to be brief otherwise I would write a neverending post :sweat_smile:)

From my side, I work with openEHR models and platforms and tools for 3 years. When I started, it required a lot of time to read the documentation from AM and RM models, understanding the structure and create my own archetypes and using the archetypes from ckm, creating templates, learning how aql’s works, and many many others, but I enjoyed it so much that I decided to do my master thesis about openEHR. I have some of these templates from many projects already deployed on production and I would say that 90-95% of them are using CKM archetypes. Currently my main work is doing the full stack per se, if we can call it in that way - archetype, templates, forms, clinical terminologies, aqls/views.

I have been working on many models lately - which I will be soon sharing with the openEHR ckm - and as I said I always try my best to use archetypes from CKM or from national instances to do my templates - even if that takes me more time to implement. If in any case I need to create a new archetype I always try to share it with ckm.

With this, I had the opportunity during these 3 years to check other models (many of them deployed and in production) that were made all around the world by many people on different companies. What I have been realising is that many of these models can be done using purely the archetypes available from CKM but they haven’t and most of the time the reason is “we don’t have time for that research or to construct that”. OR archetypes are downloaded from ckm and internally changed (not even specialized) and added on templates and deployed that way - usually the reason appointed is “we didn’t know we could not do that”. With this then there are problems on the AQLs because it’s not getting what was expected. I could point many other cases but these two could be a good start for a discussion.

I see many people doing models without having the sufficient knowledge to do it (although this can be quite subjective depending on your level of experience) - I also didn’t had it when I started and I keep asking and studying/learning everyday - but I feel that using modelling tools like archetype and template designer can transform something that it’s not that easy into something that seems to be a “piece of cake” - we can see currently many models around that are not really that sharable (interoperable?) since everything has been changed in a not correct way or not created nicely (archetypes and templates).

What are your feelings about this? What would you suggest to fix it? Did you saw this happening too?
I am afraid that a great technology that is so useful and it’s turning big and well known can be stabbed slowly with these cases (I guess we could - more or less -use the an analogy from FHIR with different versions and everyone tweaks it a bit here and there, not following the specifications and then it’s different everywhere).

PS: i don’t exactly know what’s the best place for this post - on clinical or development - i see them as a whole matter for this topic.

2 Likes
2

Hi Vanessa

I am relatively new to actually authoring and maintaining clinical content models and definitely see this as an issue for us too. It is essentially about governance of the organisations making use of openEHR. This is impossible to police. The technology is still relatively new and people are still finding their way with it - I think this inherent flexibility is more of a benefit than a hindrance. People will inevitably build content to meet the specific local requirements, and that is ok. I think convergence will happen as organisations come to expand the use of their content or expose this externally to others.

This is not really something that I think can or should be mandated - it has to be an organic process of convergence over time and the openEHR approach and associated tooling, like the international CKM, allows us to do this.

So in our organisation I always go looking on the international CKM first for any models I need - this makes a lot of sense - but we also need local content and sometimes changes to the CKM models for our purposes. If these changes are sensible for the international community then we should feed these back in, though to be honest I have not worked out how we do that yet.

Maybe what we need is some more blogging / wiki best practice for governance of openEHR CDR content, to help guide new users to a more consistent way of doing things. Documentation is not great, in general, for openEHR but then it is up to us in the community to try and correct that.

Anyway, I think it will probably sort itself out, but some guidance documentation on best practice would probably help that happen sooner rather than later.

2 Likes
3

Hi Paul,

Thanks fo your reply. I was thinking that probably was only me seeing this kind of problems since I don’t see it mentioned anywhere, unfortunately. But it’s really nice that we can start to speak openly about this issues - I guess this way we can figure out best practices and “how to” with everyones experience and find a solution.

I fully agree with you:

  1. This should be governed, probably a team responsible within companies to govern all the content deployed in different projects would be the best - I also have suggested this before on my thesis and on many instances while checking theses problems. And this team would be deciding which internal content should be sharable on CKM and sending content (There’s also national CKM instances, I have noticed that only Norway, UK and international are quite active, but the same does not happen with the other cases, seem abandoned?)

  2. Official documentation from openEHR about this - due to no documentation what usually happens is that new implementers take advices from more experienced people as non-true or correct since there’s nothing saying that they should not do things in one way or another, since it keeps “working” - many of them are pure programmers without any clinical thinking of how things should be done. What I mean with this, and I will give a very basic example (could give many many others), is having a vital signs template using a composition archetype (e.g. encounter) and with 1 archetype only that has BP (only having datapoints for systolic and diastolic as quantities), body temperature (only having temperature), etc - there’s no clinical concept per archetype. All of this could be done purely using CKM archetypes. Who created this is aware of consequences? Because, in end of the day, “it just works, why should I change it and complicate it so much”.

  3. I understand that local content is needed, it happens everywhere and in many cases openEHR CKM does not have all the necessary content because things cannot be generic to everything - I also create my archetypes when I don’t have a archetype from CKM that fulfil my use cases and share it after. And it can be slow to follow all the new content being done internally. It also takes time for reviewing, but if we don’t start to share models, what’s the point? Having the models as internal only - I see that as using them as new business model. I see way more worrying about platforms than content - a platform without content does not do anything by itself. I defend that content is way more important - but that’s my very honest opinion.

If any use-cases or suggestions are need for documentation, I will be very glad to help.

2 Likes
4

Dear Venessa and Paul,

This is an important topic and concern - it’s good to see it being raised here.

I’ve searched back through my document and email archive since day 1 of openEHR, to locate a closely related discussion I initiated a long time ago. At that time, there probably wasn’t yet enough wide-ranging practical implementation experience of openEHR clinical modelling methodology, to inform an initiative to collate and document emerging best practice. But as Venessa’s post emphasised, we are now at a point where the concern needs addressing. Incidentally, Venessa, I’d like to read your thesis that you refer to, which sounds very interesting. Is it available somewhere online?

To give you an idea of our thoughts at that time (seven years ago, so well out of date, now!), I will attach below the email enquiry I circulated to colleagues within openEHR, in August 2012. Apologies that this makes this post very long.

I have kept material collated from the about ten responses I received to my original enquiry. It didn’t get any further at that stage but I’d happily send it to you if you plan to start a group to work on the topic now. There’s too much to place on the forum, here, and I’m not sure how much would still be relevant.

Other openEHR members, like you, who are dealing with these issues on an everyday basis, will have a much better informed sense than me about how such an initiative might best be framed today, should there be the wish and wherewithal to try to take it forward within the now very much wider openEHR community.

Again, many thanks for kick-starting this topic.

Best wishes,

David Ingram

<<< < < email enquiry August 8th, 2012

It has been on my mind for some time that the experience of implementing openEHR archetypes in real world context is being gathered by numbers of key people, like you, at a very important formative phase of the evolution of health informatics discipline. Aligning the theory and the practice of two-level modelling needs to be kept under careful review, to help us create corporate real-world memory of this experience. It is also rapidly becoming a pivotal issue in persuading others to join in the journey, and thus move the field towards a tipping point, that I am sure is coming.

If one pauses and takes a step back, as I have the luck and privilege now to be able to do, one can see that there is already considerable and invaluable experience of the life-cycle of design, implementation and on-going update, curation, governance and support, of ensembles (groupings) of archetypes and templates, from multiple standpoints – clinicians, computer scientists, medical technologists, health care managers, companies, universities, professional bodies.

I have recently been thinking and discussing about an initiative that I would like to kick-start, to help draw this rich experience of openEHR people together, for the Foundation. Knowing that everyone doing useful things is having to run incredibly hard, just to keep going, I have come up with an idea for a pilot experiment, to test out what might usefully be done.

This letter is to sound you out (is it a good idea, how can it be improved, would you be interested to help me?), and, if you are interested and able, to invite you to join me in this, as one of a small group of key people I am asking to give up to two hours of their time (ie a largely off-the cuff response, reflecting what is on the top of their mind), and writing no more than one side of A4 to capture this personal perspective, in bullet points and max. one or two sentences of justification, for each or any of the following set of seven issues (suggest improvements, please, if this off-the-cuff set seems unbalanced or sub-optimal in some way, to you).

  1. Modelling a set of archetypes and templates for a health care institution
  2. Creating and sustaining an institutional openEHR-based infrastructure
  3. Modelling a set of archetypes and templates for a clinical or health care domain
  4. Creating and curating an archetype and template repository
  5. Determining what and how to model with archetypes, templates and terminology
  6. Determining ‘optimum’ granularity of the clinical models
  7. Deciding whether to get involved, as a systems developer and provider, in shaping products and services around openEHR

So, you might, for example, write one or two bullet points on each of the seven, or seven or so bullet points on one of them, or any mixture thereof.

I know this is deceptively hard to do, and I don’t want slow thinking about it; it’s fast thinking that matters to me at this stage (cf Kahneman (a Nobel prize winner), ‘Thinking fast and slow’).

Can you first let me know, as soon as possible, whether you would like and feel able to help me, and also make any suggestions for refining the set of seven issues. I’ll then review and recirculate an updated request for you to frame and submit your entries. As it’s holiday season, I’m not sure how long this will take, but I’ll keep you posted about progress. As and when I have final replies, I’ll then distil the answers, report back, and consult with you all about how to publicise and seek wider engagement, if that looks to be a good idea.

I hope it works out as I am sure it could be an important step.

2 Likes
5

Hi
First of all, thanks to Vanessa to start this dicussion. Again, I must say, because the topic has been discussed in various settings a number of times before. Both at Slack (openeherclinical.slack.com) and in Twitter. (The Slack discussions disappear after some time, due to limited space for storing).

It’s correct as Vanessa states, some pure programmers will see the archetypes as some variable they can define locally, just as in any programming language. And it works. And if the goal is to just get the damned system to work, that’s all right! If the data isn’t supposed to be shared with other systems, we do not need to standarize. But it will stay in it’s silo, and there is a great risk for other programmers to define the very same type of information, or something partly overlapping, in other archetypes within the same system. Not good. Besides, it’s re-inventing the wheel over and over, and that’s something we like to tell the world openEHR helps mitigate.

I see the problem with limited resources in the developing teams. And also the time allocated to find and reuse the existing archetypes. And especially it they’re not perfect for the use case. As a project manager I’m shameful myself for putting bad archetypes into production due to shortening of time to go-live date (or no more funding). There’s a balance between a pragmatic approach and the perfect solution regarding internationally published archetypes. And it’s painfull.

Even so, we have to move away from the situation where there is little or no feedback to the community of altered archetypes in production. The changes done can be perfect, but they can also be based on misunderstanding (or not understanding at all) of what the clinical concept is about (BTW, that’s why it is extremely important to document in the archetype header section - to describe how the archetype is supposed to be used in non-technical or non-clinical wording).

There’s possibilities to give feedback to the community in the international CKM. Use the discussion option or post a change request to any archetype. And if you’ve already made a change to a archetype - upload it. Same if you’ve made a new one - upload it as a candidate.

Given this, and if the international editors have time (and funding) to digest the incoming postings - we will hopefully be able to support the community with a more and more broad and diverse package of published ready-to-use archetypes.

Cheers,
Vebjørn Arntzen
(Editor of Norwegian CKM)

2 Likes
6

Thanks for a really interesting and important post, Vanessa.

I completely recognise the issues you raise but like Vebjorn and Paul, a fairly relaxed about the current situation, although I agree better guidance would be helpful.

I have been fortunate to work on a very small number of openEHR projects where the client was able to make a ‘clean start’, was willing to listen and take my advice, and was able to fund the extra effort required to ‘do things right’.

However, as @varntzen says, that is often not a reality for many projects - timescales are tight, budgets is limited, there is a mature’ legacy’ environment that needs to be integrated, or frankly, widespread data-sharing is not an immediate priority. I often think that the core skill that I bring to these projects is knowing when to push to do things right and when to ‘do it dirty’. It would be nice to share some of that experience but it is actually quite hard to pin down because of the huge range of settings, requirements and environments that openEHR is being used.

Having said that, I don’t think we should be too hard on ourselves. We can do better in terms of helping implementers better understand the governance processes, make it easier for people to feed back changes etc but actually quite a lot of that does exist e.g in CKM. With all the flaws in our processes and throughput of change requests etc, I am not aware of any other health IT clinical standardisation methodology that is working better and delivering at the scale we have done, especially given the essentially voluntary nature of most of the work.

To some extent we are still at the mercy of a market, which in theory calls for standardisation/interoperability but actually still largely often prefers to deliver local datasets when the rubber hits the road.

We are just at a stage where many people (and I have been there) get it a bit wrong the first time round! Better education / training/ funding will help but to some extent folks just need to ‘own’ the chaos a little at first.

2 Likes
7

Vanessa:
Many thanks for your email to my UCL academic account, responding to my post above, and for the link to your dissertation, which I have speed-read through this evening. It’s a very creditable piece of work - the quite complex project you succeeded in implementing demonstrates a considerable capacity and talent for teamwork, on which openEHR has always utterly depended. The results you obtained in comparing local and international CKM repositories helped in clarifying the specific concern you raised in this topic. If you are happy to do so, I’d recommend posting the link here. There are a few points from the thesis that I’d like to follow up on - can you send me an email address for correspondence, please?

2 Likes
8

Hi David,
I am very glad to read this :blush:

The thesis is open and online at https://repositorio-aberto.up.pt/handle/10216/119499
It was made in 2018. With the following experience I got after it, I would add even more important things to it. But I guess it’s nice idea for start.

I will send you my email by message.

@varntzen and @ian.mcnicoll thank you so much for telling your experience. I think is very important to have all these aspects being phrased here - it can help the new implementers to understand what needs to be done or at least taking always that into consideration. We all have been in that little chaos in the beginning and that’s normal, even more when we don’t have someone mentoring and telling the reasons of why should we do A and not B. The problem is when we have an entropy of mistakes and misinformation due to:

  1. not enough time to do research and do things properly - i think the following success will be dependent from here,
  2. keep doing local definitions because there’s no understanding of what’s the difference of using international archetypes, what’s the RM types allow to do (where using an action, an observation, an evaluation, etc - EOD everything is a observation or a cluster) and it’s “quicker”,
  3. keep having many new request of projects because of having the “openEHR” name on it, which does not speak for itself since it depends if many aspects are being well done (by what we have been speaking on this topic) but for what it aims to.

Once again thank you for sharing your experiences. Would be great if this topic keeps growing with more examples and solutions.

2 Likes
9

Hi all,

This topic was discussed at the openEHR Board meeting today.

One idea that was suggested was that we hold regular web-based ‘clinics/seminars’ where clinicians, informaticians and tech people can share their experience, practice and approaches to various common modelling scenarios and challenges. Possibly building a discussion around a short ‘show and tell’ by one of the more experienced modellers in the community.

Would this be helpful?

Ian

3 Likes
10

Hi
Yes, @ian.mcnicoll especially if there is a possibility to raise questions ahead of the online seminars. This will help newbies (and others as well, even the more experienced modellers learn almost every day for every new archetype built :grinning: ).

Unfortunately, I don’t think this will solve every problem Vanessa raised. We need also to reach:

  • Developes, who mistake archetypes for “just another variable”

  • Project managers and decision makers to understand the need for, and plan for activities for (possibly) making new archetypes

  • Customers, to understand the value of using standarized archetypes, and not to accept “bad ones”

In short: We need a online school! And eventually, a sort of certificate for modellers/developers/vendors. “openEHR certified”.

I know this have been discussed before, and with limited resources in openEHR International, it can be hard. Any billionaire is highly welcome to sponsor this.

Vebjørn

2 Likes
11

I feel we probably need a ‘Bad Archetype’ logo- drawing on the ‘Bad Robot’ theme

image

3 Likes
12

Dear colleague,

For many years I thought and wrote about modelling artefacts in health and care.
There are several topics that must be addressed and agreed; topics that deal with modelling using archetypes.
Without these agreements generic interoperability/interpretability is impossible.
A commonly used stabdrd on Archetype/Template modelling is the missing link in health-IT.

Some topics are:
1- Models for any documentation, observation, inferences, planning, actions, processes, clinical treatment model, use of classifications/terminologie, context meta-data, etc.
2- Kinds of uses of information inside any EHR-system (de novo information, re-use of information
3- Modelling style (specialisation by changing the name/meaning of nodes in an archetype versus specialisation by changing the content of leaf-nodes)
4- Agreement about what classifications and terminologies to use for nodes inside archetypes

I collected my thoughts in notes that constitute 150+ pages on Archetype Modelling. It needs rewriting, editing and updating, but I’m willing to share it and improve it.

13

Hi Paul,

Totally agree with your approach here.

Start with CKM archetypes wherever possible. Specialise or use CLUSTERs to extend existing archetypes for local requirements. Build a new archetypes only where necessary and, preferably, aligned with existing patterns, and hopefully submit as a proposal to CKM.

If you do identify new requirements that are not just local (ie jurisdictional/administrative) then it is likely that others will need it at some time too, so submitting a change request will potentially contribute further to creating a maximal dataset. Remember, that our intent is to be inclusive as much as is practical and sensible, so even published archetypes can potentially continue to grow. Publication of an archetype does not mean ‘final’ or ‘finished’, but only designates it as fit for use as per the peer reviewers at the time of review. Governance processes will allow for enhancements/changes to be made and even errors to be identified and archetypes to be deprecated and replaced if patterns are incorrect or unfit.

It has always been our intent to publish more about modelling but given that it is rather ‘bleeding edge’ the domain has been evolving. I think we are in a position to document more, and we have been slowly.

In the meantime this forum can be a place for asking advice re patterns or other archetypes that might form the foundation for new ones.

I am certainly hoping to get back to some blogging about modelling. I started early on, but haven’t had the capacity to do so in recent years. It is a hope of mine to return to this in 2020 - we’ll see how it works out in reality… :slight_smile:

Regards

Heather

1 Like
14

Hi Vanessa,

Totally agree with your observations about modelling, development, consequences for implementation etc. It is the reality that we live with, warts and all.

However there is no magic bullet here. My approach has been to keep soldiering on with the development of high quality, well researched archetypes, slow as it has been. And with 100 or so of core reusable archetypes we are getting to the point where they are being well used.

I’ll take any small win as a step in the right direction and acceptable in the short term…

We know that the archetypes are being referenced and used in many ways outside openEHR implementations. Even if the archetypes are not directly implemented in openEHR systems, at least the data points might be consistent, and so we then contribute to interoperability by minimising dangerous transformations.

And if only some CKM archetypes are used, and others locally ‘hacked’, at least some of the data is consistent and consistent with other implementations. Small steps, small wins. We can’t influence the pressures of individual projects or companies, which are numerous and it will always be tempting to take shortcuts in the interest of very real time and money constraints.

The reality is that it may take a few generations of developers to quit thinking ‘invent locally’ and not seek the archetypes from CKM - it has been the way they were taught and naturally gravitate to. The enlightened developers, such as yourself, have a large role to play in educating about the value and benefits, and your thesis will be a valuable contribution towards this change.

In the long term, education and demonstrating value/benefits etc of the complete openEHR approach to developers and implementers is a job for other developers and implementers. In my experience clinicians asking, requirements etc is not enough. We also need to educate procurers, project managers etc - everyone who is involved in the planning and decision-making such that the dev and implementation phases are optimised, rather than compromised.

We need a multifaceted approach, teaching principles of modelling archetypes, discerning use of information models alongside terminology, better tooling to make it easier to reuse archetypes rather than create a local, non-interoperable one etc.

But the main source of our credibility is to keep building and refining our international pool of archetypes - that is where we stand out in the digital health community. Not sure if the “Build them, they will come…” approach is enough to change digital health. But for sure the opposite is true - if we don’t build them, they definitely won’t come and participate :smiley:

3 Likes
15

Hi @varntzen and @ian.mcnicoll, I totally agree with both. The “openEHR certified” would be a nice one - as a result from a quite complex online test or a presential one in the openEHR day events, where you need to reply how you would model something based on a use case, doing an example of how would a template json would be filled until getting an aql (and many others) and then get the score - but yes it’s necessary to get money from somewhere to apply that :sweat_smile:

@GerardFreriks thanks for your reply. Could you share these notes with us? I think that would be really valuable and this forum is a really nice place to share ideas and learn more with eachother. I would really love to read them.

@heather.leslie thanks also for your reply - I also use your motto “take any small win as a step in the right direction and acceptable in the short term” on my daily basis. It’s really grateful when we see people start to understand why something needs to be done first in one way and if there’s no use case possible, then we go in a different direction. But I think we can really improve the initial modelling by having some notes/warning calls from official channels - openEHR specification page is the “de facto” ruler for every openEHR platform implementation - a few bullets there (and these bullets has been mentioned here by everyone, thankfully :grinning: ) would really help in order to be seeing it as a paradigma in archetype modelling.

3 Likes