Allow serial-group to cancel jobs from the queue
under review
V
Val Ter
It is awesome CircleCI natively implemented queue for jobs. One of the main feature of the eddiewebb/queue orb was ability to skip some jobs conditionally, so if one instance of the job is running, other instances will be skipped completely. Unfortunately, this option is missing in a new feature provided by CircleCI.
Use case is next: we have different workflows (let's name them 1, 2, 3, 4) to build and deploy different part of our application. At the end of each workflow end-to-end tests will be triggered (technically it's implemented as a call to run another workflow with tests, let's name it T). When some of workflows running simultaneously, they will call tests several times in a short period of time. Obviously that in this case we only need single test execution, not all of them.
My suggestion is to add a 'add-to-queue' parameter to a 'serial-group' key. The behavior of that parameter and possible values are provided below.
I tried to cover useful cases on the attached screenshot, in the description of the "add-to-queue" parameter. This block describes the logic of the parameter I want to be implemented.
- add-to-queue:
- type: enum
- enum: ["all", "none", "latest"]
- default: all
- description: "Parameter that defines if new instances of the job should be added to the queue
- all - when job (1) is running, and job (2) and (3) and (4) triggered
- - in this case all new instances of the job (2) and (3) and (4) will be added to the queue
- none - when job (1) is running, and job (2) and (3) and (4) triggered
- - in this case no new instances of the job will be added to the queue, all jobs (2) and (3) and (4) will be cancelled
- latest - when job (1) is running, and job (2) and (3) and (4) triggered
- - in this case only the latest instance (4) will be added to the queue,
- and all other instances between (1) and (4) will be cancelled.
- In case job (1) is still running and instance (5) triggered,
- it will be added to the queue and instance (4) will be cancelled"
Activity Feed
Sort by
Nathan Fish
under review
We will take a look at adding this as an enhancement.
F
Fernando Abreu
Thanks a lot for the feedback and for walking us through your use case—this is super helpful.
Would you mind sharing a snippet of your config or answering a couple of quick questions to help me better understand the setup?
- Are you programmatically triggering a new pipeline to run the E2E tests from each workflow (1, 2, 3, 4)?
- In your ideal flow, would you want the first E2E test run to complete, or would you prefer to cancel earlier ones in favor of the latest?
- If E2E tests are already running and a new change comes in, would you want to cancel the current tests and run fresh ones, or let the current ones finish before starting another?
Really appreciate the suggestion around the add-to-queue behavior.
V
Val Ter
Fernando Abreu
- Yes, we are programmatically triggering a new pipeline which contains E2E tests only, and we do this at the end of all workflows (1, 2, 3, 4)
- We don't want to cancel current execution. We had an experience with a big sequence of requests to run E2E tests. So if cancelled, it could lead to the situation where we don't have test results for a long period of time due to constant cancelling of tests.
- We want to let current executions to finish.
I tried to cover useful cases on an initial screenshot, in the description of the "add-to-queue" parameter. This block describes the logic of the parameter I want to be implemented. I also added the description as text, but messed with the formatting. Will try to properly format the initial request