For non-breaking OPT changes, data doesn’t need to be migrated. It works with the new minor version of the same OPT.
@Sidharth_Ramesh Your #3 says: “update template_test_name.v2.0.0 to template_test_name.v2.0.” It is missing a minor version number at the end. I guess it should be “v2.0.1”.
I believe your example of adding a new mandatory field cannot occur during a minor OPT version change (it would cause all the existing compositions to fail the validation with template_test_name.v2.0.1).
@vidi42 Your #3 says: “…Composition A.v2 based on template_test_name.v1.0.1”. Is this correct? I guess it should be “…Composition A.v2 based on template_test_name.v2.0.1”?
I believe the CDR should handle Composition A.v2.0.0 even without updating its template_id to v2.0.1. Composition A should be valid for any v2.n.m OPT version available in the CDR.
Adding a new mandatory field requires a new major OPT version. This does cause the issues you described regarding AQL. Adding a new OPT version to all AQL queries is one way to handle this (it doesn’t require data migration) and is how Alex described handling Composition B.
Another approach is to migrate all compositions to the new major version OPT. This approach seems impractical due to the volume of data requiring migration (and the OPT authors not providing a migration plan).
Neither scenario is ideal for the users. In practice only the AQL scenario is available to the clients. The consequences of failing to update all AQL queries are real. I’m not a fan of putting the full burden on the clients. I also believe it will be bad for the openEHR image.
One way I’m exploring to mitigate these issues is storing archetype data using inheritance. For example tables for blood_pressure.v1 and blood_pressure.v2 inherit from the blood_pressure table. This allows using the version-less archetype name in queries (the database includes the data from the v1 and v2 tables). When a new major blood_pressure.v3 archetype is introduced, most queries will continue to work and will include the data for the new version. The good thing is that the queries will fail loudly instead of continuing to work while silently ignoring the v3 data.
The above works only if the new major version didn’t “break” the structures used in the queries. In other cases, the CDR could try to handle it by introducing alias fields and other technical solutions to the version-less archetype table. Some cases will still require changing the queries or migrating data. I would just like to investigate how much the CDR can help with new major archetype versions at a “technical” level.
@Sidharth_Ramesh I believe you encountered this problem because you mapped RM data directly to SQL tables. PostgreSQL is just not up to handling openEHR data with that approach. This is why I picked ArcadeDB which gives us more options when transforming RM to native database “objects”. Inheritance, alias fields, and strictly typed JSON fields are a great help.