Option to allow failures in Fan-In/Out Workflow
complete
B
Brandon Page
The requires tag in workflows really limits what is possible. Requires makes a lot of sense for the deployment scenario used to demo it, but waiting for a group of jobs to complete (pass or fail) should be an option.
Scenario: Setup -> Run multiple sets of tests in parallel -> Combine test results/coverage results/artifacts and report to PR
That scenario above isn’t possible because if a single test fails in any of the jobs running in parallel the the fan-in step won’t run.
CCI-I-344
F
Fernando Abreu
The changelog for this feature is available here: https://circleci.com/changelog/more-flexible-job-required-capabilities/
You can find the documentation here: https://circleci.com/docs/configuration-reference/#requires
Nathan Fish
complete
This has been released. Documentation and change log entries coming shortly.
S
Sherif
any status on this feature? our org is checking this page daily. we need a job to run after parallelized tests are completed regardless of their outcome.
Nathan Fish
Sherif: We are working on rollout now. Should be released within the next few days, assuming no surprises or bugs found.
Vikram Krishnamurthy
Hey, we could also use a clean up step in our flow when dealing with cleaning up held deployment locks during "fan in". Or some kind of "finally" step that happens no matter what at the end
seaona
hi there! What's the status on the Requires solution proposed by Nathan?
P
Peter Darton
Another usecase is CircleCI orb testing.
At present, it isn't possible to write a CircleCI CI build to fully built-and-test a CircleCI orb because it isn't possible to "check that a command or job will fail in <situation>" as part of the automated test.
A circleci orb build needs to test that the orb code functions correctly, and that means calling the code and checking that it works as expected, which requires you to check not only "does this work when it should" but also "does it fail when it should" too.
I want to be able to "run this (downstream) job regardless of whether or not the previous (required) job passed or not" so that my orb build can check "did that work? If so,
actually
fail the build for real because that job-under-test should've failed".The easier you can make that, the better.
We would also benefit from this "run this job even if the previous one failed" facility for test-result combining (which we want to do but can't do as-is), but I would've thought that the inability for CircleCI to do CI build-and-test of CircleCI orbs is a bigger issue.
The difference in these two usecases is how the workflow's status is calculated though.
In the case of test-combining, we'd still want the workflow to be declared an overall failure if a fanned-out test failed even if the test-result-combining job succeeded.
In the case of orb job testing, we'd need the workflow to be declared a success if the expected-to-fail job failed and the workflow to fail if the expected-to-fail job succeeded.
TL;DR: We want to be able to say "this job failing doesn't mean the workflow ends here" somehow ... but sometimes we need the workflow to declare a failure and sometimes a success.
Nathan Fish
Interested in everyones thoughts on the following workflow graph when using "Requires". Does it give you enough information to understand why a job ran or did not run?
Robert Peralta
Nathan Fish Wow that is perfect! This is definitely self explanatory imo and solves the case I'm trying to address.
F
Fil Maj
Nathan Fish looking pretty good! Does one get similar information if you hover over the '3 jobs not ran' branch? I assume those were not run because they require job1 to pass?
Nicholas Shaw
Nathan Fish the second image seems to show it requires the prior job to fail, is that correct?
I think the main use case is the automated tests example given, whether the job passes/fails, I don't care, I want a test report generated.
Instead of needing to do things like the following:
- run:
command: |
if [ "$TESTS_PASS" = true ]; then
exit 0
else
exit 1
fi
Nathan Fish
Fil Maj yes that's the intent.
Nathan Fish
Nicholas Shaw in this example that is scenario, requiring failure. If you didn't care about the job passing or failing you could certainly model that with requires with the "completed" keyword. See the example config below for more details on what that would look like.
E
Ezequiel Gonzalez Rial
Nathan Fish In the first screenshot it shows that if a task fails, the workflow will fail. Wonder if there is an option to make the workflow success?
This is something that has been discussed as CCI-I-646
Nathan Fish
Ezequiel Gonzalez Rial at this time you can't override the job status. We have discussed internally how we might meet that need. Most of the use cases we have discussed are directly tied to events reported to github checks. With requires you can create job dependencies regardless of the status by passing both success and failure.
Nathan Fish
in progress
F
Fil Maj
Nathan Fish yay!
Nathan Fish
We are set to start working on a solution for "more flexible requires" and are looking for some feedback from you all. The config syntax would look something like the following:
jobs:
- a
- b
- c:
requires:
- a # defaults to success
- b: completed # success or failed
- d:
requires:
- a: failed
- e:
requires:
- b: canceled
There is one caveat. Canceling workflows does not mean the canceled job logic is reached. Cancelling a workflow still cancels all remaining work in the workflow.
Thoughts or considerations we should take into account?
Augusto Xavier
Nathan Fish Sounds like what we are looking for!
F
Fil Maj
Nathan Fish looking good so far but for clarity, could you describe the various job states that could be specified as part of
requires
? I see success
, failed
, canceled
, completed
as options but it is unclear to me what canceled
actually means? Is that a job state or a workflow outcome?Nathan Fish
Fil Maj great question! Cancelled will be a new option where an "approval job" can be cancelled, not just approved. It will be possible to cancel an approval job via API or via the UI for approval jobs.
F
Fil Maj
Nathan Fish Thanks for clarifying, LGTM 🚀 :shipit:
J
Josh Empson
Numerous teams here could really use this functionality as well (even outside the fan-in/out workflow), for example the case of scaling up a platform in order to run tests, running the tests and then then scaling down again regardless of the success of the tests.
I understand why
requires
depending on a successful completion would originally have been sufficient, and keeping this as the default behaviour would be completely fine - however the continued lack of switch/parameter to simply depend on job completion via success or failure is incredibly frustrating.Just a simple:
workflows:
deploy_and_test:
jobs:
- deploy
- run_tests
requires:
- deploy
- do_thing
requires:
-run_tests
when: always
would be incalculably helpful, and prevent the need for ugly
waiter
/ listener
jobs that seems to have become the standard workaround.With 331 upvotes on this issue, I'm obviously not alone in this thinking.
Please sort it out, Circle!
Load More
→