Account for missing test results when using "Re-run failed tests only"
N
Nivl
When tests are run with parallelism, tasks which do not complete for any reason may not upload test results.
When "re-run failed tests only" is run CircleCI retrieves the failed tests, and re-runs those. It does not re-run tests that are missing due to a failed task. This can result in successful tests being reported and tests from failing tasks being ignored, and subsequently potentially causes PRs with missing tests to be merged.
Implementing logic to check for failed tasks and running those tests along with tests reported to have failed would prevent accidentally merging PRs where tests are missing.
J
Jonathan [[Unknown]]
This behaviour - which we consider a bug - impacts us mostly in terms of integration time and costs.
"Rerun from failed tests only" is an extremely useful feature which we have been using extensively to reduce flow run times and credit usage.
The behaviour described above necessitates extra care when choosing whether to use this feature or "rerun from failed", making it prone to human error. To avoid such error, we were forced to disable "rerun from failed tests only" in our flows altogether. Hence, the impact on our cost and time.
The suggested implementation would allow us to re-enable and use "rerun from failed tests only" freely without having to worry about accidental merges of PRs with some tests not having been run (or even failed).