When I conducted a live openEHR webinar last time on Dec 19 2024, I instructed close to 200 people from around the world to download all Archetypes (export all) from the International CKM at the same time. This crashed the website and many people on the webinar were not able to access CKM. I understand that this is similar to a DDoS attack on a smaller scale, but I think there’s value in people trying out the CKM themselves.
I am conducting another webinar on Feb 28 2025. And I’m planning on instructing everyone to download all archetypes from the International CKM again. I don’t know how many will exactly show up, but 200 simultaneous downloads / exports will be a decent estimate.
I want to request the team at Ocean and openEHR international to implement some sort of caching / performance improvement to handle this number and let me know if I can go ahead and direct everyone to directly download on the CKM.
The backup option would be for them to download a zip file from Github or Google Drive. But it’s not the same impact as them being able to download from the CKM.
Not sure who I should tag , so I’m including @sebastian.garde @Seref @ian.mcnicoll
Please let me know before Feb 28 if you want me NOT to use the CKM directly.
Thanks!
2 Likes
This is a very much a decision for Ocean and there possibly is an argument for throttling CKM from inadvertent or deliberate DDoS but I don’t feel it is reasonable to expect CKM to perform at this sort of scale routinely for what is a very artificial situation.
We no longer do a full download of archetypes from CKM. We would always select them explicitly or clone from the Github mirror., and we make a to use of the AD option to import from CKM directly via the ‘Update repo’
I can understand the ‘marketing’ impact of what you are trying to do, but in all honestly, I would show people the download all option, but then explain why 200 people doing this at once is going to cause understandable issues. Point them to a link where they can download .zip or better still, create a .zip that reflects your training example, rather than the full set of archetypes.
4 Likes
Hi @Sidharth_Ramesh First thank you for the advance notice.
The openEHR CKM server is hosted by openEHR. It may be possible to give it more power (not Ocean’s decision), but for typical high use it is running ok I believe.
The way it is, I am proud if you say that 200 people got to the point that they could load CKM more or less at once, even if the subsequent downloads did not work. This is well, well beyond typical high use as you can imagine.
On top of that the Download operation is probably the most expensive of all the things you can do, so I am certainly not surprised that this does not work for 200 people at once. Even with a considerably more powerful server, this will be a tough call.
We can throttle more of course (as we do - so far in a lenient way - for the API), but that of course would be not much help with having 200 people download at once.
It simply has not been a practical problem so far because typically it is CKM’s automated export to GitHub that is used for this purpose since we set that up.
I am not sure what you are doing but downloading or just looking at a few selected archetypes individually and then go to GitHub to download all if needed sounds useful too.
That said, we can certainly look into better caching for this particular operation (it is more difficult than you may think though), but I need to understand what exactly it is you are doing there. I’ll DM you if you want to setup a quick call to discuss further.
6 Likes
@Sidharth_Ramesh use CKM-mirror instead! Github is better at handling high loads and likely do cache things (something for CKM to consider, just invalidate cache of zip when content is changed). Direct link to the highlighted choice below is https://github.com/openEHR/CKM-mirror/archive/refs/heads/master.zip share that with your paricipants instead.
Teaching people to download the zip from CKM is just a plain stupid antipattern. That is not a good way of working with archetypes, forking CKM mirror and work in suitable branches in your fork is much more sustainable, especially if more than one person collaborates. That’s how we do things in Sweden at national level and in Stockholm https://github.com/regionstockholm/CKM-mirror-via-modellbibliotek/
5 Likes
Thanks for all the input. I think at this moment it’ll be best to just give a tour of the CKM and then download archetypes from the Github mirror. Thanks @sebastian.garde , @ian.mcnicoll and @erik.sundvall for the advice!
4 Likes
