DV_PROPORTION.is_integral in OPTs and validation rules

Hi all, I’m working on testing a server, and found for DV_PROPORTION a constraint for “is_integral” ends up in the OPT: https://github.com/ehrbase/ehrbase/blob/develop/tests/robot/_resources/test_data_sets/valid_templates/minimal/minimal_action_2.opt#L247-L273

By the spec, “is_integral” is a function, not a field, so that constraint looks strange there. Is that a bug of the modeling tools?

Looking at the spec definition for is_integral(), the description is a little confusing: “True if the numerator and denominator values are integers, i.e. if precision is 0.”

For that to make sense, I think when ‘precision’ is mentioned, is the precision of each numerator and denominator, not the precision of the DV_PROPORTION. But if that’s referring to the DV_PROPORTION.precision, for that function definition, we need to consider numerator, denominator, precision, type and the validation rules, because are all related. The way I think that should be wirtten is:

if precision is 0, then numerator, denominator and type should be integer, integer, [fraction or integer fraction] to comply with the validation rules (I didn’t include ‘percentage’ because I think we could have values like 33.5%).

What do you think?

And everything is interlinked because of the validation rules:

  • precision = 0 implies is_integral
  • is_integral implies (numerator.floor = numerator and denominator.floor = denominator)
  • (type = pk_fraction or type = pk_integer_fraction) implies is_integral

Testing Marand’s implementation, I see they also use is_integral as a field for validation:

"Invalid value at /content[openEHR-EHR-ACTION.minimal_2.v1,'Minimal 2']/description[at0001,'Arbol']/items[at0002,'Proportion']/value/is_integral, expected: not true, actual: true"

Should all functions in the RM be treated as fields for validation?

I would say better to not do that, as there are times values are derived from actual attributes or compossed from several ones. This is something you probably cannot do with xml schema (probably is feasible with schematron or rules)

Yes, but this case is different than value derivation: the OPT actually contains a constraint for something that is a function, not an attribute like the rest 100% of the RM. I found strange this doesn’t happen anywhere else so I think this should be a problem of modeling tools or the RM representation those modeling tools use.

For instance in the BMM the is_integral is there as an attribute, even though says “is_computed”.

Checked that “is_computed” attribute and only appears 3 times in the whole model:

I guess something “is_computed” when 1. it is an attribute, 2. it’s value derives from other attributes.

But is_integral() is a function not an attribute https://specifications.openehr.org/releases/RM/latest/data_types.html#_dv_proportion_class

Maybe my assumption about “is an attribute” is wrong, or maybe the model/modeling tools have some bug there.

Anyone can confirm?

Analyzing the validation rules for DV_PROPORTION there is one that is unclear, here a summary from the spec:

precision = 0 => is_integral()

type = ratio (0) => any(numerator) && any(denominator) && any(precision)

type = fraction (3) || type = int_fraction (4) => is_integral()
is_integral() => is_integer(numerator) && is_integer(denominator)

type = unitary (1) => denominator = 1
type = percent (2) => denominator = 100

denominator != 0

If my interpretation of the spec is correct, then: if precision = 0 => is_integral() => is_integer(numerator) && is_integer(denominator)

Which I don’t think is correct: we can have non integer numerators and denominators, e.g. we get data from a sensor or lab analyzer, but we want that to be interpreted with precision=0 for the proportion, for instance: 2/0.5, which is a valid ratio.

What do you think?

The first two are functions, but ‘most_recent_composition’ is something that doesn’t exist in the specifications at all. It is in there by mistake (from some experimental version of the RM that I vaguely remember from long ago). So we can remove that one right away. The other two should be removed, but we’d need to know if there are any archetypes mentioning these fields. @sebastian.garde do you have a way of checking this?

2/0.5 would be type pk_ratio (0) as far as I can see. I don’t understand how that could be interpreted as having precision = 0…

Lists about 10 archetypes that have is_integral

I am pretty sure this was a ‘feature’ of the Archetype Editor - Archetype Designer does not add is_integral, it just sets precision to 0

I’m pretty sure we could fix the current CKM archetypes as a patch change What do you think @sebastian.garde ?

“precision” is a field, so the value 0 can be set by a client, even though that makes or not any sense. If that doesn’t make sense, do we need an extra rule?

I think the template designer actually puts the is_integral there, but might come from the AE as well. IMO that shouldn’t be there since it’s a function, complicates data validation.

I’m not able to test properly with Template Designer but I know that is_integral() was introduced by AE - there is a checkbox 'Integral 'that adds is_integral to the ADL. I’m pretty sure that TD does nothing more than pass this through when it generates the opt.

Historical artefact / bug - arguably the .opt generators should ignore and not pass-through. I certainly agree that it should not be there.

1 Like

Thanks Ian, I detected the issue because it was affecting data validation in EHRBASE, which I think uses Archie at some point. I’ll do some testing in AE+TD to know more.

You can also just search for is_integral in CKM directly (using the full search).

A long while ago, there have been long debates about this one as well as EVENT.offset and - to a lesser degree - PARTY_RELATIONSHIP.type.
It is also why in the Java Ref Impl Validator there is special code for these three to indicate that it is a “commonly constrained functional property”.

While I agree that functions should rather not be constrained, the questions remain how to model that it is integral or the offset is 5m if you cannot constrain them.

It would probably be reported as a minor version change, but you can overwrite that if need be.

That seems like an additional rule. I will go further and say that if it is something that has been constrained already in some archetypes it should be changed to a rule and remove the constrained function in the archetype.

Hi all, I just faced the same issue again while updating test data in the conformance framework for HiGHmed.

I’m not 100% sure if I could manually remove the is_integral constraint from the OPT to be able to proceed with my test case or if tooling should be fixed. Besides AE passing the issue to TD, I think the TD should be able to check the RM for validity of attributes vs. non attributes, though I know maintenance of the AE/TD is slow and not sure if/when this can be available.

That is one of my questions, if the DV_PROPORTION.precision applies to numerator and denominator or if it applies to numerator/denominator. But from your comment, I guess precision applies to each numerator and denominator, so when both are integers (precision = 0) it implies is_integral() = true (which is one of the invariants I didn’t totally get).

Though, what I mentioned, is “precision” is another attribute which value could be set by a client in a JSON/XML message. But in the RM, since numerator and denominator are Real, it is difficult for a program to check if the values are integer, because if Real is mapped to a Float, any number will be an approximation, having 1 != 1.0. So I see the need of “precision” as a separated attribute.

For the RM 3.0 I would suggest to create different data types for ratio, proportion, fraction and percentage :slight_smile: which would remove the need of the precision, is_integral, etc, etc.