CircleCI Ideas

Sequentially run jobs in a branch

In some instances it may be that a user makes multiple commits to a branch in a short amount of time, causing multiple Workflows to execute at the same time, possibly deploying in an incorrect order. "Auto Cancel Redundant Builds" may not be suitable for all situations in which multiple deployments are expected. 

Solution: Enable a way to ensure that commits to a new branch are queued for sequential execution.

  • Avatar32.5fb70cce7410889e661286fd7f1897de Guest
  • Jan 24 2019
  • Taking votes
  • Attach files
  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    29 Mar 16:46

    A related idea that this would seem to address is that two half-finished workflows are worth a lot less than one finished workflow.  Even without deploys, I would rather have CircleCI prioritize _finishing_ a workflow that it has started over _starting_ a new workflow.  Then, instead of hanging around waiting for complete results for two different pieces of work, I can wait for less time for one result and then act on it while CircleCI is producing the second result.

  • Admin
    Kunal Jain commented
    10 Apr 21:35

    Thank you so much for your interest in this feature. Initial straw-man that we have been discussing is to have serialization at a pipeline level or rather at workflow level. If pipeline/workflows are serialized, commits would build the order they were received by CircleCI. This feature is definitely on our radar but is currently backlogged behind some refactoring work. Happy to reach out once we start working on it. 

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    07 May 01:43

    This problem actively discourages people from increasing their concurrency (paying circleci), I'd think it would be a huge priority.

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    10 Jul 13:14

    This is a big issue, since even the workaround like https://github.com/eddiewebb/circleci-queue are having problems. With that workaround there is still an edge case with workflows on job transitions not being seen (which is probably an API subtlety, since the API only lists jobs, not workflows)

     

    Neither can we build workaround with external locking, as there is no way of having a "on fail" trigger to clear the lock, so heavy timeouts need to be used - which then always block builds after failed ones.

     

    Any other known workarounds here? Or an update of this issue?

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    06 Aug 19:10

    We could really use this functionality to aide in our build/lint/test/deploy pipeline. Current work arounds to "Stacked Deploys" include Orbs like https://github.com/eddiewebb/circleci-queue, which suffer some race conditions. But they also mean a Workflow could be running, but effectively queued because it's in a `while true; sleep` loop. A built-in feature to the CircleCI platform to enforce serialization ought to be able to avoid the "running, but effectively queued" scenario.

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    23 Oct 16:34

    This is an essential feature for us. The alternatives supported natively in CircleCI are not sufficient because they result in risky race conditions.