a first step using an algotithmic conversion of legacy source system to templates based on integration archetypes containing e.g. GENERIC_ENTRY modeling the source system’s structure (so that we can kill the old system without losing data)
a second step with manual mapping from the templates based on integration archetypes to more standardised templates based on international CKM archetypes (and with a link from those COMPOSITION instances to the corresponding integration-archetype based COMPOSITION(s) that may not always be 100% complete or have a 1:1 cardinality correspondence)
Questions:
what tools/methods have others used for step #1
what tools/methods have others used for step #2
Some of this we explored in GitHub - regionstockholm/poc_tc2openEHR · GitHub but we might want to try something more sophisticated and controlled than string matching/mapping to FLAT paths this time. We are also for step #2 considering adding some features to our IntEHRgrator mapping tool, but it needs to get more stable and usability hardened in its initial form first…
Hi @erik.sundvall did you consider having archetypes/templates representing the legacy data as-is and then map template (legacy) to template (new model)?
I’m working on something that detects changes in templates and proposes a data migration path from the old template to the new one. This is still very green but I see potential.
@pablo, if I understand you correctly, I think what you are proposing is exactly what I described as step 1 & 2 above - if you by “as-is” mean “as-data-is-structured-in-the legacy-source”.
I’ll edit the text a bit above to clarify even more that #1 is a first step.
We might not neccesarily want to logically remove (version-overwrite) the integration archetype based Compositions, in case somebody really wants the details as they were in the old system. Thus not really migrating/version-upgrading a legacy/integration composition to a CKM-based one, but rather creating a new composition with a link to the legacy composition(s).
We plan to flag/mark legacy different than the nicely standardised corresponding data so that you can choose what you want to returned when querying.
A clarification on my side, what I meant was without GENERIC_ENTRY, so creating archetypes with normal entries but with the shape of the legacy data for the first step.
@pablo What you are suggesting is doing step 2 directly, just using your own archetypes instead of the international ones. The problem with that direct mapping is probably not in the selection of the RM classes, but in finding the right attributes of the RM for each class and data items. That still would require a deep analysis and specific mapping definitions, and what I understand from @erik.sundvall is that this is precisely what they want to avoid right now. First, they need a quick and automatic normalization of data (or migration to an openEHR platform), while complete normalization will come later, as it requires more time or resources.
Starting with that Step 1, something essential is to keep enough metadata about the source of each data (the original tables/fields, schemas…) in the feeder_audit. That will be your most valuable source of truth about the original semantics and context of each data item to succeed in step 2.
this is similar to how we are doing it with the HIP Integrator (HIP Integrator: AI-assisted health data integration – vitagroup HIP). The difference is that our approach is to create / generate templates using AI which mixes existing archetypes from CKM with new and bespoke integration archetypes based on type generic. So we are doing a single step mapping. However, I think it really depends on project goals and preferences.
@damoca I think what @birger.haarbrandt is actually doing the deep analysis and specific mapping definitions with the help of AI.
I do think for the data extraction you still need to do a lot of work, how much depends on many factors, the main one the complexity and consistency of the source data. The real issue is that the ETL work needs to be done, you can do a very dummy and quick extraction, but then the “TL” parts will be more complicated. So it’s not about doing it fast, is about how to distribute the complexity in the 3 steps E, T or L.
In what I’m thinking I agree there is a big effort on the extraction, but smaller on the loading, in which AI can be of great help. But the whole ETL work is unavoidable IMHO.
I’m also thinking of the tools we are designing right now: having a migration plan over two versions of the same template could be automated even if there are breaking changes, instead of having two totally different templates in place and integrations/mappings between them need to be defined. It’s also easier and could be automated even more if there are many template versions in place with small breaking changes on each version, so we can gradually morph the source info into a totally standardized destination. Anyway, this is all in the “lab” right now, we are testing this thing…
This also includes the “E” part of ETL by helping define views to extract the data. This considers finding and using IDs for patients, encounters and data to define the boundaries of compositions . The question is if there is so much merit in the “first pass” when mapping to a pure integration template. We provide instead a “raw ingestion zone”/ bronze layer where tables can be synced using CDC. Maybe you win some metadata by mapping on a template first and you could use AQL, but the actual use in clinical applications might be limited as you would build against a temporary data model.
We probably need some more information from @erik.sundvall to really understand the motivation behind this strategy, I’m just trying to respect what he said:
so that we can kill the old system without losing data
And I’m making a lot of assumptions here:
I assume that the main priority is to have a single technological stack based on openEHR ASAP, and do not keep the original sources. It is somewhat similar to the bronze layer that @birger.haarbrandt mentioned, but using openEHR itself as the data repository
I assume that this migration only aims to keep that historical data safe, and will not support a running legacy system that is generating new data. Otherwise, adapting that system to work with the “plain” openEHR representation, and later doing a second adaptation of the system to the “formal” openEHR representation, would only double the work.
I assume that there might be some legacy data items that do not fit in the existing archetypes and will never be normalized, but they can be kept in that transitory openEHR representation.
So, I’m not arguing against going directly to the complete normalization. Doing it in two steps means doubling the risk of errors during data transformations. However, a two-step approach can improve data lineage.
It looks like some of you are suggesting going from source system straight to #2 based on properly CKM-archetype based templates (do full ETL at once). That is a valid way for some data too, it is the “green” variant in the TakeCare to openEHR PoC described in presentations and videos linked from GitHub - regionstockholm/poc_tc2openEHR · GitHub We used that e.g. for the medications API that does not contin a lot of structural variation.
In this thread i am actually asking for tools/methods for the “purple” way in those presentations. E.g. for the legacy documentaion module woth over 20000 “keyywords/(headings” used in thousands of different forms over more than 20 years. It is not realistic to do do the fill T & L of ETL for all of those before we shut down the old system, thus we want to time-shift that and T & L work going from “purple” to “green” and just do the mosts important from that documwention module to start with. We will make a tool that allows the clinicians to browse the “purple” part too, not just the “green”.
Both ways (green/purple) ar valid for different use cases and in the TakeCare PoC we used both. So thanks for good intentnions but you do not need to spend energy on explaining that the “green” way is also an option. WHat i am asking for is if anybody has used the “purple” way and what methods and/or tools you found useful.
On my side, what I had in mind was to use the violet way, autogenerated templates, but not with GENERIC_ENTRY, since that generates an extra modeling debt you need to solve later (so Extraction becomes easy, but then TL become more complex).
Adding on top of that one or more versions of the template, making the actual “transformation” of the source data into several templates that have breaking changes between them (template changes and versioning become the Transformation steps).
Then migrating the data from the current template version into the next (data that complies with template v0 needs to be transformed into data that complies with template v1, hopefully, without losing data or meaning), and making these migrations automatic or semiautomatic.
This differs from the green in almost everything, begins with an alternative the violet phase, and engineers the path between the violet to the brown, making the final version of the template the actual “CKM” template (or whatever you choose as the final step for direct ingestion into the destination).
I’m not saying this is better than any other approach, it’s just what we are testing right now. Hopefully we’ll have a paper about this next year.
