I believe my developers are wasting time and money because of flakey tests -- they rerun a workflow from failed because of a single red test case, and we re-run thousands of previously green tests which eats up CircleCI credits. I'd like to understand re-run behavior; it would be lovely if Insights could collect this data for me and display it!
Warning Status Code
My team is integrating performance metrics into the build and we would like the ability to have a job with a status of warning that turns yellow instead of just having the success/fail statuses so that we can warn developers when performance metrics are getting close to thresholds.
I want to know which tests are failing the most across all branches, not just master. It's nice that Insights shows the most failed test in each branch but 99% of the time what I care about is which tests are flaky when I submit a PR. CCI-I-785
approval in a command
we have very large config,yml up to 2k lines and if approval type is describe as a command we can wrap the jobs top of it as a reusable
Metrics that track shipping speed
The metrics in insights are currently all focused on the workflows and jobs that Circle runs. Metrics that help me track my team's shipping speed would add a ton of value. Metrics could include: Time to review: the time it takes for a GitHub PR to be reviewed after it is opened Time to prod: the time it takes from when a branch is started to when the branch is merged to master and gets deployed Number of review cycles: stats about the number of rejections a PR gets There are lots of interesting stats here that would bring me back to the Circle dashboard even when I'm not trying to figure out why my build broke.
Link to failed tests
When looking into failed tests from the Top 10 list, I want to be able to drill down into the samples of the failures. As it works now, I have to go click through all failed runs to find one that contains the failed flaky test that I'm investigating. If the test is failing 40% of the time, then I've got a 4 in 10 chance of finding that failure by browsing through failed runs.
Avg time of steps in insights
It would be nice to get info on step timing across builds. CCI-I-505
Show resource consumption per job
Being able to see resource consumption by category (e.g. CPU, memory), per build and job would be extremely useful when debugging issues. If this was available via the Insights API or UI then we'd rarely need to manually re-run jobs via SSH to inspect state and work out what had occurred.
Ability to add custom insights / stats in project
It would be great to have an option to push custom data into insights. This might include build size test suit time run repository size num of files within the repository .... basically anything that devs want to keep track of CCI-I-436
metrics for datadog: max queue time, median queue time, current queue depth
So CircleCI Insights's dashboard shows "Median Queue Time" -- but: does not show what is the period over which that "median queue time" was calculated (was that over the past hour? past day? past week?), and does not give me the ability to configure CircleCi Insights's to calculate this over a different period (maybe I want to see over the past 2 weeks the median queue time... It would be great if Datadog has that "median queue time" so I could graph that information as well as set up warning and alert thresholds. I could also use: "maximum queue time" so that I could monitor the longest a test execution has had to wait for a container -- for instance, if the maximum queue time measured over the past 60 minutes shows that one job had to wait 5 minutes for a container...maybe it's time for another container "current queue depth" -- for instance, if the queue reaches 5...maybe it's time for another container Thank you for your assistance! CCI-I-1135