Dynamic config (aka: Setup Jobs)
complete
A
Alan
Circle 2.0 has been great, but our team would like to be able to be able to control how the build pipeline operates at build-time by dynamically generating the circle.yml.
At build, we'd like to be able to somehow run a command that dumps out the content of a circle.yml. By doing so, we could calculate a build plan on our monorepo for the workflow that should be run specifically for what projects in the monorepo have changed on the branch. This would make it easier for us to run jobs in parallel and kick off a build flow that's much more dynamic.
CCI-I-184
Salil Subbakrishna
complete
Salil Subbakrishna
Hi all, wanted to share an update here. We just started the open preview for scheduled pipelines, our new scheduling solution which is compatible with dynamic config. Preview docs are here: https://discuss.circleci.com/t/scheduled-pipelines-are-here/41684
C
Caran Chris
I was working on implementing dynamic configs in a repo and the blocker around scheduled workflows has stopped me dead in the water.
Liya Ai
Caran Chris: Hey Chris - just making sure you saw Salil's note below. We're working on an update that should resolve your blocker.
C
Caran Chris
I saw, thanks.
Nicolas
Liya Ai: Do you have an estimated timeline for that fix release? Thanks!
Salil Subbakrishna
Hi everyone, wanted to provide an update here. We are currently working on some dynamic config features. At a high level, these changes will allow customers to define a “Setup Workflow” which has jobs that can be used to set config and/or pipeline parameters, which will be appended to the pipeline. This is currently under development and we expect it to be available in the first quarter of 2021
Chenglim Ear
Salil Subbakrishna:
Hi, I've started trying out the new dynamic configuration feature and it's great. It seems that it doesn't work with schedules. Can you point me to information on how I would dynamically set up a schedule using the dynamic configuration feature? Thanks!
Salil Subbakrishna
Hi Chenglim Ear, right now dynamic config doesn't really work on schedules because we manage and initiate schedules differently from webhook/API pipelines. You may have noticed that webhooks/API trigger an entire pipeline while schedules are defined at the workflow level. We are working on an updated version of the scheduler that will make schedules more similar to webhooks/API calls, which should make dynamic config work with schedules.
Justyna Janczyszyn
Hi Salil Subbakrishna: are there any updates on this? My team switched to using setup workflows recently and only after the transition realized the scheduled workflows don't work. We'd love to have a workaround for this as soon as possible :)
Salil Subbakrishna
Justyna Janczyszyn: Since this is a complete rebuild of our scheduler, we are expecting this to be available in the next quarter.
Justyna Janczyszyn
Salil Subbakrishna: is there any temporary workaround?
J
Jeremy Klein
Salil Subbakrishna: Hi Salil, is there any update here now that the next quarter is almost done? This is a pretty big drawback to using dynamic configs for us since we rely heavily on scheduled jobs to push nightly builds.
Salil Subbakrishna
Jeremy Klein: We've been working on our new scheduler that should solve this issue, you can expect an announcement about it in the next few weeks.
J
Jeremy Klein
Salil Subbakrishna: cool, glad to hear it! Thanks for the quick response!
Pieter Van der Haegen
Salil Subbakrishna: Is there already a more concrete timeline? or a way to beta test? This feature would really help us a lot!
Salil Subbakrishna
in progress
L
Liya Ai
under review
L
Liya Ai
Merged in a post:
Programmatic pipeline generation
M
Mkobit
As a developer, I want the ability to generate my pipeline at runtime so that I can customize the stages and steps before execution. The idea being to "run this thing to generate the configuration (or config.yml) instead of hardcoding everything".
A few reasons I think this is useful:
1) Semi dynamic pipelines - not fully dynamic like Jenkins Pipelines, but more flexibility compared to now with what I would say is not much complexity for users
2) Less templating - IMO templating in YAML files is confusing. Thanksfully, there isn't much now but I would be dissapointed.
3) No need to invent additional YAML features. GitLab CI recently introduced the include directive, which I think is a massive mistake.
4)
One of my use cases is to build against multiple different platforms with fanout and fan in when those versions are not hardcoded into the YAML file.
CCI-I-249
j
j s
+1 I need this!
N
Nandreev
Any update?
L
Leo Yu
+1
agreed with guest above that buildkite does this really well. The ability to generate configs at runtime would make our monorepo circleci configs SO much shorter.
Load More
→