Tooling support for inviting reviewers

Clinical
1

Hi, everyone! Happy 2022!

This should possibly also be posted in the Tooling category, I’m not sure how to do that :slight_smile: .

In the review process in the CKM I’ve encountered some issues that has to be discussed. I could have just asked for a change in the CKM functionality, but would like to get input from a broader audience first. So here it is:

Normally we invite existing users, and match the domain and experience the users have given when they registered, with the requirement the editors (and for Norway - the editorial board) think is appropriate for the archetype in question. This is normally OK, as long as we have enough users with the required skills.

However, for very specialized archetypes, there is not many (if any) of the registred users with the detailed insight in the medical domain the archetype is about. The editors then have to rely on input from clinicians when the archetype is made - either in a existing project or by gathering a team which will work as a group. This is how the group of archetypes related to genetics were made.

Having such a team is nice, as the group can be trained to understand both how archetypes should be modelled, and how the CKM functionality works when coming to the review process. But leaves a danger of the same persons gives input in the making of the archetypes are the same as reviewing (and accepting) them.

It is hard to find new users, and make them register in the CKM. Within a team or project we can persuade and guide them. But it is far much harder to persuade clinicians and experts not familiar with the openEHR thinking to give feedback. It can be several reasons. Many are just too busy and/or can be reluctant to register because they are afraid of getting spammed. They might not have any direct use of the archetype in a EHR, maybe not ever. They can be afraid of being held accountable for the use of the archetype in a clinical setting.

All this leads to problems finding people to give a proper and reliable peer review.

One example: In a ongoing review of five archetypes related to Artficial reproductive medicine (“In vitro fertilization”) we have only two or three reviewers - one of them being one of the editors, another is a dentist. The invite has been sent to 125 users!

The Norwegian team has sometimes solved this by presenting the archetype for clinicians not registred in the CKM, either in a meeting or by sending a pdf/Word version by mail, and summarize the comments in one of the editors own review, and documenting who made the comments for each element and in the ‘Overall comment’. This is a hack.

I propose a new functionality in the CKM where an editor can send invites to not-registred users by generating a personal link to the archetype review. This link expires after deadline of the review. The user will be regarded as an guest user, and can only be invited manually for each review by editors in the project the archetype is owned by. It will be a task for the editors to either reject the review or accepting it. If accepted, the name of the reviewer will be visible (as normal).

This way we will be able to both have control of who participate in the reviews and also be able to collect input from clinicians or experts who are not able/not interested/too busy/too afraid to registrer themselves as users.

Thoughts?
@natalia.strauch @joostholslag @ian.mcnicoll @heather.leslie @siljelb @sebastian.garde

Kind regards,
Vebjørn

3 Likes
2

Hi Vebjørn,

We do need to register people because there is not usually just a single contact. Often there is more than one review for a project etc. And there are many reasons we need the data, to discern how many people are reviewing and that there are not duplicates, to ask a question of the reviewer etc etc. Registering once simplifies so many governance things; removing registration will require all sorts of governance hacks or CKM reconfiguration.

In reality, there are few things worth doing that don’t require a simple registration and validation of email. After all this is participating in a review of clinical data standards, so is not unreasonable to expect this as a minimum.

It is helpful to understand better what the barriers to registration so that we can mitigate the issues.

  • Do we need to revise/streamline the registration process.
  • Is CKM too difficult - maybe use the project-related invitation line sent by email to potential reviewers and providing access directly to the project eg Clinical Knowledge Manager
  • Is it too hard to understand how to participate in a review…

Any insights?

1 Like
3

I agree with Heather’s assertion that we do need to register people. But at the same time, the fear of being spammed is understandable, and to a certain degree well founded.

Currenly reviewer availability is a binary on/off choice for each user, and global for the whole CKM. Would it be possible, and make this better, if a choice was added for users to state that they’re available for reviews, but for specified projects only?

Edit: Maybe the existing “reviewer” role in projects could be used for this? Each user could choose to be available for all reviews, or only for archetypes owned byin projects where they are registered as a reviewer?
image
(also, this choice should be moved to the first page of registration, with the user being directed in wizard fashion to choose their domain and profession, and potentially the relevant projects, if the first or second choice is selected).

2 Likes
4

The trade-off with making registration so specific is it requires or assumes an understanding of projects and reviews etc.

If the registration process for a newbie is already potentially confusing/convoluted/difficult ie a turn-off in any way then this will only add to the level of difficulty.

Maybe we need to separate the processes:

  1. The simplest registration would benefit just getting people to put a foot in the door.
  2. Then follow up with an onboarding process or series of emails that includes fine-tuning options to specific domains, professions, projects etc.

Any suggestions from recent CKM newbies?

5

I agree. My suggestion would be relevant mostly where people are specifically recruited for very specific projects, under close editor support.

I like this idea. Maybe, when you’ve done the simple registration, you’d be immediately asked to respond whether you’d like to set up your detailed preferences right away, or do it at a later point. Then, if you choose the latter, the CKM could ask you to set these preferences at a later point like it does now or similarly.

6

I think Vebjørn has a really good point here. I do understand the objections from a governance point of view, but this is about the user perspective and their needs as reviewers. It is a barrier to reach enough of the right people for reviews, so changes are needed that support both end-users and administrators. Also, not every reviewer has an account today either, they gather around someone that has an account and do group reviews and let the one having an account sending in the comments. You have even less control over those reviewers than you would have if you could use the solution that Vebjørn suggests. My guess is that being able to participate through a temporary link lowers the threshold to register an account the next time. It is quite natural that people are doubtful (I don’t think I am the only one getting irritated when being prompted every now and then to create accounts for peripheral services that I maybe only will use once).

3 Likes
7

Of course, we could set the system up to a light touch email invitation to a reviewer, assuming we fully understand the reviewer requirements and there is consensus on ‘light touch’ requirements. The reviewer could respond and we have another review. Yay, I guess.
BUT

  • What if they want to edit their review?
  • Or want to make a comment.
  • Or add a change request.
  • Or…
    Then they need a login. Adding a login to edit a review after the review has been submitted is not trivial, especially for the reviewer who has multiple email addresses etc. Or we’ll end up with even more duplicates. It is not simple to loosen, or sidestep, the registration process - there will be consequences. Sebastian will likely have many more use cases where it will create difficulties for CKM.

I don’t see a problem with group reviews in principle - it is a common part of the learning curve or internationalisation. The main difference is that it still functions as a registered user - meaning the review can be modified, change requests can be added, discussion questions asked and there is a login/registration that can be utilised as required by Editors eg contacting the registrant/coordinator to ask questions or seek clarifications. We may miss out on the detail of the reviewers but this can be mitigated eg the NO reviewers are always added to the international archetype.

With CKM we have built a community of volunteers and a philosophy of donation of expertise and time for the greater good. It concerns me that if someone can’t/won’t take a small amount of time/effort to register, then it may impact the feedback. Not sure how we quantify this, but common sense implies that if there is no sense of commitment (as opposed to intent or goodwill which may be abundant+++), then there is often the same amount of care about responding or the quality of the feedback. A light touch invitation certainly does not guarantee more participation or better feedback.

If someone wants to investigate a ‘light touch’ CKM option, including pros/cons, proposing new specs, quantifying them (dev $$$) and providing an estimation of the benefit, I’m certainly willing to explore it further.

In that context, consider the possibility of adding a (supposedly simple) CKM function to email a review request to an individual’s email. In turn, this prompts registration prior to commencing the review. @varntzen recently wanted to invite people to 5 archetypes simultaneously, so what should he do - invite them all to the first one, await their registration and then invite to the remaining 4 etc. Or have 5 registrations, one per archetype, for the same person. Or if no registration is included and they want to modify their review… It quickly becomes an Editorial/governance nightmare.

For now, I’m of the opinion that we need to make it easy for all reviewers to register (and manage/finetune their account over time) rather than cater for outlier situations. The best option for all is to work to identify the barriers/problems to participation and fix them for everyone.

2 Likes
8

I remember from signing up to CKM I found it not too frictionless, it was exciting and I was looking for a way to contribute. But others who come for a more onetime experience it may be off putting.

I myself also struggle to accept review invites because form the invites it’s hard to judge wether I’ve something meaningful to contribute. (At least my own assessment is I usually do :p)
One of my thoughts go out to expanding the information and the interaction in the email. So print out the archetype, reviewers questions etc in the archetype. And ideally also allow the reviewer to respond to the email with their comments.

I think we may be able to get those answers for the scenario where a specific ‘unregistered’ reviewer is invited by an editor on the basis of their e-mailadres. We could even let editors create accounts if that would make it easier.

There’s also worlds to win in making it attractive to register I’m thinking marketing/sales funnel type improvements. Do we have any people with a background in that area, or someone with experience at companies that do this very well, like booking.com?
There’s probably also improvements to be made in just the text by better explaining what a registered user can expect. No specific ideas at the moment but I’d be willing to walk through the setup process with someone to collect ideas…

I share your concerns Heather. But in the situation vebjorn describes a single low commitment review is already very valuable imho. And more importantly I think the low commitment may grow into high commitment after a good experience. How else will a reviewer decide he will invest more? So I’d like a system where an email only ‘account’ is enough. But registering later gives added benefits (e.g. editing review and the other concerns you described) if the user registers with the same e-mailadres, it van even be the same account. Dev costs may be prohibitive indeed. But once we see an approach that may grow, we can start very small (e.g. let editors create accounts for reviewers).

2 Likes
9

The email text is indeed very general, and I wish it could be a mix of the general part and something from the archetype or the specific invitation text in the archetype review. For example inserting text from the Purpose section in the archetype, and then go on to the ‘Your options’ part of the email. At least this will help the invitees to understand the scope of the archetype.

This is a possibility, but might lead to a long and overwhelming email. I think a link to the CKM is sufficient, as today.

This is in fact possible, and the super-duper adminstrators are also entitled to activate the user without the user’s confirmation of the email address (and also the user’s knowledge). It’s meant for helping the user, when/if there are trouble in the registration and confirming process, and shouldn’t be abused. If someone made me a user, and invited me without my knowledge, I would certainly be surprised, not to say angry.

In the Norwegian team we’ve thought of various ways to encourage the existing users to participate in more reviews: To celebrate the top ten reviewers (but they are always from within the editorial group :laughing: ), paying per review (but have no money to pay), having a annual lottery among reviewers (but can’t buy anything to give), competition between user groups/regions/professions (but is unfair, because there are not a fair share of archetypes relevant to all professions), bribing (no, not really). If someone has good ideas both to premiere new users and reviewers - please come with them!

Sadly, this is true. And there are also other improvements, more important ones, to be made first. We have to prioritize.

I’ll give comments to the others in separate replies.

1 Like
10

Hi everyone

Thank you for a good discussion and valuable input! (and it’s not over yet).

I’ve been aware of the possibility to share a link to the registration dialogue from within a project, and this leads the user back to the project again afterwards. This was new to me, and is an oppurtunity we should use.

Yep, I agree. My intention with this thread was to start a discussion and try to figure out how we can improve things. We’re closer, but need to identify in detail how the user’s journey from a unknowing new user to a passionate frequent reviewer can become easier.

The step-by-step approach as promoted by @heather.leslie and @siljelb above should be persued further.

11

The project-based sign-up link looks like a good option to explore for this.

If the user accepts the invitation to the project as a reviewer (either by signing up to CKM or also via a project invitation if the user exists already), the user can then be invited as part of inviting all members of the project (with a certain role) to the review round.

I agree that “reviewers” of a project should be able to be invited to reviews of archetypes/templates in that project. In my view, it would be fair to assume that if a user accepts the role of “reviewer” for a project, that this user is happy to do reviews in that project. (Appropriate wording and functionality changes are required in CKM to make this happen and clear). Making this part of the user’s project role sounds more straightforward to me than adding an explicit third option to be a reviewer in selected projects only. This combines Silje’s and Heather’s views on this (in my understanding of your comments above) in that it provides the functionality without requiring an explicit additional choice.

Account creation by editors
Editors can create accounts directly, and there are use cases for this (maybe less for the int’l CKM instance). To clarify: When an editor creates a new user, a welcome email is sent to the new user. So this is not stealth mode, however obviously should still not be used to create user accounts if the user hasn’t agreed to this. It may be useful where editors have talked to the prospective user beforehand.

Invitation texts for reviews:
This has been raised by Vebjørn previously and I agree that this should be implemented, so that additional details can be provided in the reviewer invitation email sent to the reviewer.
A really long time ago, we used to include the complete invitation text (as displayed in CKM when opening the review round) in the email as well, but this became too convoluted, especially when the instructions got more detailed over time. But a shorter and motivating custom text added to the generic invitation email and ideally geared towards the archetype at hand, may be very helpful indeed to increase reviewer uptake.

Generally making it more satisfying to contribute
I really enjoyed Vebjørn’s list of ideas on how to make it more satisfying to contribute.
There may be other ways to acknowledge & celebrate users that have contributed a lot in CKM directly or outside.
The annual lottery for some kind of voucher maybe where people can participate if they completed at least one or maybe - say - three reviews over the last year is a nice idea from my point of view which may be implementable without too much money if openEHR is so inclinced. I don’t think anybody would register to CKM for that (and if so the user’s motivation would be rather dubious I believe) but it may be a token of appreciation for longer term commitment.

3 Likes
12

Thanks for your thorough response, Sebastian!

I agree with your reasoning that project-based reviewing is probably easier than adding another global level option. However, there will likely be at least three levels of involvement that we need to facilitate:

  1. Those who want to participate in any review regardless of project
  2. Those who only want to participate in reviews within a certain area, in this case defined as a project
  3. Those who don’t want to participate in any reviews at all

Currently we have functionality for 1. and 3. We’re discussing adding something for option 2., but how do we do that without removing option 1?

1 Like
13

Agree this will be useful, but I think it, if implemented, needs a functionality to invite users to new projects.

14

I would keep #1 as it is now and merge the new #2 into #3.

#1 is usually based on your interest as advertised via the ontology (but may differ depending on how this handled in CKMs)

#3 is what you get when you just sign-up for any CKM but without being a member (in any role) of any project.
This of course assumes that it is fair to assume that if a user is a project member (e.g. a user with the reviewer role in that project) of a certain project that this user can be invited to do reviews for resources of that project.

Users can either be added directly to a project by an editor or they can be invited to the project and have to accept before they can become a member. Users can also request to be a member of a project and editors need to approve. Project Invitations and project membership requests have rarely been used to my knowledge (and probably not very prominently displayed in CKM) but the functionality exists in principle.

15

In practice, what you’re describing is the majority of people signing up, without clicking the box about reviews availability at all? I was thinking of them as a subgroup of #1. For #3, I was thinking about the people who actively uncheck the box, so the CKM will prevent us from inviting them.

16

Any new user is asked on the first real login if they want to be a reviewer and if so are guided to select their health domain etc. They can also postpone the decision and “Decide later”, then they will be asked on the Dashboard on each subsequent login until a decision is made.

As long as they haven’t made a decision, they can be invited if necessary - the main reason for this is currently that new users invited to a project to review etc. can be asked to review immediately.

This is the status quo. Depending on your point of view, those that haven’t made a decision yet are a subgroup of #1 at the moment (they can be invited), albeit none that have actively expresed their willingness to participate in reviews (it is unsure if they want to be invited). Also they likely haven’t filled in any of the helth domain etc information either).

However, I am unsure if this should remain that way if we allow project members to be invited to review resources of that project even if they don’t want to review otherwise.

  • A - Willingness to participate in any reviews explicitly expressed
  • B - Willingness to partipate in reviews of “my” projects only
    ** B1 - Not a member of any projects → No willingness to participate anywhere
  • C - Not decided → Could either be treated as A or B