Maintaining tests for API's can become a heavy task depending on how many API's and routes you have to maintain and how often things are added.
One could also think of offshoring the testing to a dedicated company. I gave it a try and received offers which calculated a fee for each route to test.
Unfortunately, none of the companies I contacted bothered going into details what kind of tests they will deliver.
I use the API first or design first principle to outline the requirements of the API under development and do this by defining it with OpenApi specifications.
I wrote an article some time back of the tools I use to write OpenApi definitions.
The requirement
Mostly I was interested in contract testing. If we think of the OpenApi defintions we wrote in contracts, we want to test if that contract is honored by the actual API endpoints.
I am expecting that a particular endpoint returns me an object, a user for example and that user has required properties, "first_name" and "last_name" for instance.
So I could go now and write that test in the test framework of my choice. Job done.
But tomorrow I will have a new endpoint to write tests for and I also want to verify the request body I send to POST and PUT endpoints meet the requirements.
If you have been following my post about writing OpenApi specifications you should have seen the prism mock server from stoplight.
Under development you can use prism mock in proxy mode which will mark everything which does not comply with the OpenApi definitions of your API.
So if we have our API definitions, why not generating tests from it?
I found and testsed two tools which do exactly that for us. Huge time saver!
We have some additional requirements though, we want to give our developers a tool to use while developing.
We also want to run the exact same tests in our CICD pipeline. If we fail the job if the API does not full fill the requirements, a good part of the job for the quality assurance is already done and documented!
Adressing the problem
I few months back I started to play around with schemathesis. Schemathesis allows us to auto generate tests from our OpenApi definitions as described above.
To use schemathesis, one can use the docker image and run it locally for verification.
The docker container can also be used in the CICD pipeline, mostly GITLAB on my projects.
A stage of a gitlab job looks like this with schemathesis:
test_api_contract:
stage: api_testing
image:
name: schemathesis/schemathesis:stable
entrypoint: [""]
variables:
REVIEW_DOMAIN: $CI_ENVIRONMENT_URL
REVIEW_BASE_URL: api/v1
script:
- schemathesis run --checks all openapi.yaml --base-url=$REVIEW_DOMAIN/$REVIEW_BASE_URL --workers=auto
It works well and you can check the pipeline for errors or run the container locally. The minimal requirement for developers is to run a local container to test before they commit.
While I was researching for solutions I came across another interesting project which promises another approach.
Unfortunately, there was no docker container to use in the pipeline and I didn't want to install the NPM package to the gitlab runner (pipeline).
However, in the meantime I crated a docker image myself and I plan to share write a post of the full life cycle of auto creating a postman collection from the OpenApi definitions and how to auto generate postman tests with it.