Queue parallel builds to deploy and test in sequence
As a software engineer
I want to run end to end tests for each change
So I can quickly identify breaking changes
There are 5 repos to build, for services 1-5. Services 1, 2, 3 and 4 have a large number of inter-service tests; service 5 is standalone.
Upon building code for service 1, 2, 3 or 4 in Circle CI, the build enters a step that deploys the build to an integration test environment (ITE), and triggers the full suite of tests. If more than pull request for Service 1, 2, 3 or 4 triggers this step at a similar time, the deploy to ITE is FIFO queued pending a successful outcome for the currently running integration tests.
Upon building the code for service 5, the build and deploy waits for any other Service 5 builds to pass tests, and then it deploys and runs the tests.
If Service 1, 2, 3 or 4 fails tests, their queue is paused until the problem is fixed (a successful rerun of that branch's tests) and then the builds continue.
If Service 5 fails its tests, its queue is paused until the problem is fixed (a successful rerun of that branch's tests) and then any other Service 5 builds continue.
Ideally builds should be able to run independently to a point (so unit tests can run and Docker images can build) but then should enter a sequencing queue.
Builds with interdependent repos could be gathered into Build Groups for easy configuration of sequencing.