Scenarios for change type "deleted"

Hi I’m trying to define a set of rules for a logical delete commit and have some gray areas that I’m not sure of.

1. commit after delete flow

[creation v1] => [modification v2] => [deleted v3] => ?

Can a modification/amendment v4 happen after a delete?

This is one of those cases that forks in the version tree can happen, since v2 is deleted by v3, but v1 can be forked and a commit of modification or amendment can happen on that branch.

I’m considering the delete only affects a VERSION, not the whole VERSIONED_OBJECT.

Can a delete logically delete the whole VERSIONED_OBJECT?

On a lineal versioning scheme, I’m not sure if an amendment can happen after the delete. because the semantics of lineal versioning is I’m versioning v3 not v1 if I do a commit after v3.

I think if a delete happens, that is like killing that branch, so no new versions can be added.

Is that correct? What do others think?

2. delete on persistent compos

Looking at the specs, it is not clear to me how a delete would affect a persistent composition.

This is an open question.

This is also related with the previous case, since if a version is deleted on a persistent composition, there will be more commits for it after the delete, because it is persistent.

3. delete a non-trunk version

[creation v1] => [modification v2] => [amendment v3] => …

Let’s say there are many modifications/amendments for a compo, can be persistent or event.

What happens if a version in the middle of the version tree/line is deleted?

Can that even happen?

What happens with the created versions after that?

In my head those versions created on the branch of a deleted version should also be deleted if deletes are accepted in the middle of the tree. The other rule can be only accept deletes on the latest versions.

What do you think?

Until we have a clear criteria for commits after delete, I’ll use this:

  1. lineal versioning (no branching)
  2. “deleted” compositions can be “undeleted” using “modification” change type, since we don’t have an “undelete” change type
  3. if current latest version is “deleted” the composition won’t appear on query results

You would be able to import this structure into our implementation and it would work the same way.

Pieter

Hi Pieter,

What structure?

well, a deletion is a deletion, logically. In a non-versioned system, the data object would literally disappear. In a versioned system, such as we have, the top version records the fact of deletion, which logically applies to the whole versioned object. it could, if in system A (say in hospital A), the Composition is logically deleted but in system B (clinic B) a copy of that Composition (say medications list) is modified with a new version; then if this modified Versioned object is copied back to system A, system A will see a branched tree, which communicates the fact that hospital A thinks the information is no longer there, but clinic B does have a version of this information. THis can only be resolved at the organisational level - i.e. by agreeing what is really going on with the medication list (or whatever the info object is) between the two HCFs. But - is there any real scenario where this could actually happen? I can’t imagine System A really wanting to delete the medication list Composition. that’s what it logically means doing. well as you point out, things might be ambiguous if two systems modify copies of the same versioned composition, and one of them does a logical deletion. We probably need to think about this more carefully and update the documentation. I at least would need to get a better understanding of the problem first i.e. real use cases. Well, deletion should really make the persistent composition act as if it wasn’t there, except for queries in earlier time windows. So I would say a priori that further commits should be prevented, but - maybe someone is going to say: what if the deletion was an accident? Maybe there needs to be a lifecycle operation ‘undelete’ (or ‘undo delete’). that can’t happen - a deletion can only happen as the top version. If we were talking about Git or some other typical system, deletion makes it look like a file is actually gone, and subsequent commits will create a new file of the same name, but whose version lineage is disconnected from the previous one. We could add something to the spec to force something like that behaviour. - thomas

I meant data using the criteria you outlined.

Regards,

Pieter Bos

In a versioned system there is no status for “deleted” necessary inside a composition. The system itself marks the composition deleted. With this in mind it seems to me the semantical meaning of the inside “deleted” status is meant for something else.

Bert

I shouldn’t write in the middle of the night, while travelling. Of course the deleted status is in the version, not in the composition. Sorry for confusing reply.

Have a nice day
Bert

It's potentially not a completely wrong idea: it might be worth thinking about a 'deleted' marker on the VERSIONED_OBJECT<T> type itself. As i noted before though, i'd like to get a better idea of real scenarios where the current model of deletion doesn't work properly before doing anything.

- thomas

about a ‘deleted’ marker on the VERSIONED_OBJECT type itself.

That seems a clever idea.

Because the whole version history would become out of sight. That is how it is used anyway. It is also like that in Github. If a file (object) is deleted from the repository it seems like completely vanished. Also all previous versions of the object.

Of course going back in time makes it again visible and also the previous versions.

My two cents.

Two data collections are involved: Patient registry, Patient Health Record.
I focus on the Patient Health Record.

0- Committed data is never physically deleted.

1- Inactive. ‘Copy of data’.
EHR with patient records.
The patient moves to an other place and chances Healthcare Provider.
A copy of the EHR is sent to that HcP.
The original becomes inactive and can be consulted for legal, administrative purposes, only.
The Patient record is labelled: inactive, read only for legal and administrative reasons

2- Inactive. ‘Removal of data’
The patient demands by force of law that all personal and health data is removed.
The EHR becomes inactive and can only be consulted for legal, administrative reasons

The Patient record is labelled: inactive, read only for legal and administrative reasons

3- Active: Updating of data by third party (patient)
The patient demands a change in the data recorded.
Data recorded after one or more events are changed.
The original Compositions stay un altered.
A new version of the Composition (with the patient as author) is created.

4- Active, Update by original author
New facts indicate that previously entered data about an event was incorrect. The mistake needs to be corrected.
A new version of the Composition is created.
Old data can be acted upon. The application needs to redress this fact.

5- Update before a Commit.
During data entry but before the Commit data can be changed always. Versioning is NOT necessary.

Gerard Freriks
+31 620347088
gfrer@luna.nl

Kattensingel 20
2801 CA Gouda
the Netherlands

Gérard has some points. A patient, in the Netherlands, has under conditions the right to require the physical removal of all data.
In that case no “deleted” marker is necessary.

Then there is the case of inactive patients. In case of a GP the law requires he must.be able to produce the patients data until ten years after last visit. Often patients just disappear without formally ending the relationship with the GP or they never had a formal relationship. Tourists for example. A GP is not interested in seeing persons in a listing which are not his active patients.

I once, some years ago, helped facilitating archiving those patients data, so they could be they physically removed from the GP information system. Also here was no “deleted” marker necessary.

The only situation I can think of that requires a “deleted” marker is when a information system is used for historical data research together with active clinical use..

Maybe a standard should not facilitate such combination of use cases in a single data storage information system.
When historical research is needed one should query the archive system.

Bert

Even when the patient wants all data to be removed, this means removeal in the context of the provision of helath care.
For legal and administrative purposes the data can NOT be removed but be available for these non-healthcare provision related circumstances.
One needs a label ‘deactivated’ (for health purposes.

Remember: Even when the patient has left the author (HcP) has administrative and legal responsabilities. He is accountable for many years because fo actions taken. He needs to be able to defend himself.

GF

Gerard Freriks
+31 620347088
gfrer@luna.nl

Kattensingel 20
2801 CA Gouda
the Netherlands

Gérard, I think you are wrong. A patient in the Netherlands can require under specified conditions total physical and logical removal of his data from a health care information system .

If you want I can represent you a link to the law - text which says so, but because that is in Dutch, and therefor not readable for others, and also not interesting for others I rather take that communication to private level instead of public discussion group level.

Best regards
Bert Verhees

But also apart from that. The discussion is if a standard should facilitate an inactive patient (logical deleted) to still remain in the active clinical information system with a special flag or that he should be transferred to an archive system. This is the question which is important to decide if a standard should define a “deleted” flag.

I think such a flag is a technical flag to organize some way of archiving data. It has no additional semantic meaning and should not belong in a standard defining semantic structures.

I do NOT dispute what patients can demand.

Under all circumstances it is the author that is accountable for his actions.
Even under Dutch Business law one needs to keep files for the Revenue Tax) offices.

What is meant by the Privacy Law is that all data must be logically be ‘removed’.
All health data must NOT show up.
All business transactions need to be preserved.
All accountable actions can be questioned in Public and Criminal law.

Meaning, in my view, that the patient record is there but unavailable for the provision of health care.
Meaning that either that record is tagged as ;inactive’ or the that the reading and writing rights are for the patient only.

Gerard Freriks
+31 620347088
gfrer@luna.nl

Kattensingel 20
2801 CA Gouda
the Netherlands

(Health) Data never must be physically deleted.
Facts happen and are recorded.
But not always readable.

Gerard Freriks
+31 620347088
gfrer@luna.nl

Kattensingel 20
2801 CA Gouda
the Netherlands

Hi I'm trying to define a set of rules for a logical delete commit and
have some gray areas that I'm not sure of.

*1. commit after delete flow*

[creation v1] => [modification v2] => [deleted v3] => ?

Can a modification/amendment v4 happen after a delete?

well, a deletion is a deletion, logically. In a non-versioned system, the
data object would literally disappear.

Logical deletions don't make data disappear, even if it's not under a
versioned system.

In a versioned system, such as we have, the top version records the fact
of deletion, which logically applies to the whole versioned object.

That sort of works on a lineal versioned system, the specs are not clear in
terms of how to handle a deletion in a distributed versioned system.

I agree delete should be considered to affect the status of the whole
versioned object, but on a branching scenario this is not so well defined
or maybe can't be implemented at all (see below ***).

The issue I see with versioning for deletion is that amendments,
modifications, etc. are defined at the same level of the deletions, and the
other modification types apply to the latest version, not to the whole
composition. Considering this: shouldn't delete be specified on it's own,
separated from other change types?

I always had doubts about using a commit to delete something, and now I'm
starting to understand why :slight_smile:

This is one of those cases that forks in the version tree can happen,
since v2 is deleted by v3, but v1 can be forked and a commit of
modification or amendment can happen on that branch.

it could, if in system A (say in hospital A), the Composition is logically
deleted but in system B (clinic B) a copy of that Composition (say
medications list) is modified with a new version; then if this modified
Versioned object is copied back to system A, system A will see a branched
tree, which communicates the fact that hospital A thinks the information is
no longer there, but clinic B does have a version of this information.

*** That kind of contradicts the "deletion logically applies to the whole
versioned object", since the versioned object is unique, independently of
which system has a copy of a version from that versioned object. On a
branching case, the delete seems to only affect a branch.

THis can only be resolved at the organisational level - i.e. by agreeing
what is really going on with the medication list (or whatever the info
object is) between the two HCFs.

Besides the delete case, that is the issue of merging branches, IMO is
always an organizational issue. We had another thread were we talked about
lineal vs. distributed/branching versioning.

But - is there any real scenario where this could actually happen? I can't
imagine System A really wanting to delete the medication list Composition.

Change medication list for X and I think we will have cases. If we don't
have cases, why do we included this on the specs?

A related issue, is if a delete happens by mistake, how can someone
"undelete" if no more commits are accepted on that branch after the delete?

I think it would be useful to accept commits, like amendment, because maybe
the data is not incorrect, but the previous delete commit was incorrect. In
that sense, when a logical delete happens on a system, a good feature is to
have a view of the deleted elements and allow users to undelete them, like
a trash bin.

IMO if delete can be executed by a user, undelete should also be executed
by a user, not by the organization.

I'm considering the delete only affects a VERSION, not the whole
VERSIONED_OBJECT.

Can a delete logically delete the whole VERSIONED_OBJECT?

that's what it logically means doing.

On a lineal versioning scheme, I'm not sure if an amendment can happen
after the delete. because the semantics of lineal versioning is I'm
versioning v3 not v1 if I do a commit after v3.

I think if a delete happens, that is like killing that branch, so no new
versions can be added.

Is that correct? What do others think?

well as you point out, things might be ambiguous if two systems modify
copies of the same versioned composition, and one of them does a logical
deletion. We probably need to think about this more carefully and update
the documentation. I at least would need to get a better understanding of
the problem first i.e. real use cases.

Rethinking that, I'm not sure if killing a branch is correct, since I see
the undelete case (mentioned above) happen if delete is allowed.

In terms of the spec, I'm worried about what the spec allows and try to
fill the blanks the spec doesn't provide. I'm not sure why allowing
something in the spec if we don't have use cases to support that. Maybe the
easiest would be to remove the delete cases from the spec and delegate that
to each implementation. Some may support it, some may support undelete,
some may apply the delete to the latest version, some may apply it to the
whole versioned object, others will support logical and physical deletes,
etc. There are too many cases to define all of them in an unambiguous way.

*2. delete on persistent compos*

Looking at the specs, it is not clear to me how a delete would affect a
persistent composition.

This is an open question.

This is also related with the previous case, since if a version is deleted
on a persistent composition, there will be more commits for it after the
delete, because it is persistent.

Well, deletion should really make the persistent composition act as if it
wasn't there, except for queries in earlier time windows. So I would say a
priori that further commits should be prevented, but - maybe someone is
going to say: what if the deletion was an accident? Maybe there needs to be
a lifecycle operation 'undelete' (or 'undo delete').

That is exactly what I was thinking of while writing :slight_smile:

But! as I said, I'm not sure if delete/undelete should be handled at the
change type level, it seems to be a concept on a different level than
create/modification/amendment.

*3. delete a non-trunk version*

[creation v1] => [modification v2] => [amendment v3] => ...

Let's say there are many modifications/amendments for a compo, can be
persistent or event.

What happens if a version in the middle of the version tree/line is
deleted?

that can't happen - a deletion can only happen as the top version.

is that specified?

If we were talking about Git or some other typical system, deletion makes
it look like a file is actually gone, and subsequent commits will create a
new file of the same name, but whose version lineage is disconnected from
the previous one. We could add something to the spec to force something
like that behaviour.

In Git, If I checkout v2 on a new branch, and there are v3, v4, ..., in the
master branch. If a file is deleted on the new branch, and I commit the
branch, the file won't be there for that branch, and master (v3, v4, ...)
won't notice about that. All issues will happen when I try to merge my
branch with master, there I need to decide if I keep or not the file that
is a difference between the two branches.

The real issue here is now to merge, not the delete itself. But, the
semantics of a delete happening on a branch should be specified.

I think most implementation won't implement merging, so this case might not
happen. But if the specs support branching, the merging process and
handling deletions on branches while merging should be specified.

Here is the a description by royal organization of medical institutions of the Dutch law for saving medical data.
https://www.knmg.nl/advies-richtlijnen/artseninfolijn/praktijkdilemmas-1/praktijkdilemma/hoe-lang-moet-ik-medische-dossiers-van-patienten-bewaren.htm

It was years ago 10 years. It is now 15 years. Dutch law is always harmonised with European or UN law if there is such law on that level.

There are two considerations.

  1. The medical institutions want an end point for their responsibilities to keep data access able.
  2. Patients have the right to clean up their history.

The medical institution is not required to remove the data after 15 years but it has the right to do that. 10 years ago I wrote software to facilitate that for the largest GP software user group of the Netherlands.

After those 15 years a patient can demand totally removal of every trace at a medical institution.

So. This to close that part of the discussion.

I did not read yet Pablo 's long reply. I will certainly do that maybe later today. I am again travelling.

Best regards
Bert

Hi All

Ideas from a clinical perspective: