Feedback about current conformance tests

Conformance Issue list
1

Hi everybody.
Duplicate here our email discussion.

On the last call, you talked about feedback on conformance tests.
We analyzed difficulties that we had met in tests run from the “develop” branch of the “EHR base” repository.
And we decided to share with you steps that should be completed to run tests and some of our minds.

  1. It seems to us that the base suite setting for easy running is too difficult (URL, login, password, etc):

  • suite_settings.robot contains many variables

  • some variables duplicate the variables from sut_config.py

  • sut_config.py also contains many different suites of variables

    Maybe we need to make one simple sut_config.py in which will be all required parameters for tests running (similarly *.properties in other program languages). In suite_settings. the robot will be the import of sut_config.py only without duplication of parameters. So users can make easy test running:
    download project → open sut_config.py → input variables in one place intuitively → run tests

    For example, we made suite_settings.robot and sut_config.py in our demo brach.

  1. For running most tests is needed to upload OPT. But the repeated upload gives the wrong status 409. It’s uncomfortable for local debugging. Also, we can’t use the default “Delete All Templates” keyword (about it will be below). Some thoughts:
  • allow to update OPT in OpenEHR specification REST API, currently, 409 is returned
  • design ADMIN REST API for OpenEHR specification which can delete any entities (include OPT). And then we can change the current “upload OPT” keyword. For example, at first, delete the template and then upload it again.
  1. Created of one vendor tests can run with some errors to another one. For example, we may look at our last pull request:

  • there were some errors in our templates examples

  • there was also one problem in the EHRBase system

    The same situation can be in case of a new vendor will download conformance tests and run them on his system. Maybe we need to develop some rules in this case. For example:

  • create tasks in the public Jira project for discussion

  • mark these tests by definite tag

  • …

  1. And we still have one very important question about cleaning the testing system. EHRBase solves it by restarting their docker container and jar file or deleting all templates by SQL queries. But other vendors can use different technologies stack, DBMS, DB structure, etc. So this solution isn’t universal.
    Maybe, could we use a universal script clean_sut.py which includes the required method clean_system()? And other vendors should implement this method for their systems.
    If the test should run on the clean system you need to add the “Clean System” keyword to the suite setup.
    For example, our system supports work with multiple tenants so we have implemented:
3 Likes