CircleCI Ideas

Dynamic config (aka: Setup Jobs)

Circle 2.0 has been great, but our team would like to be able to be able to control how the build pipeline operates at build-time by dynamically generating the circle.yml.

At build, we'd like to be able to somehow run a command that dumps out the content of a circle.yml. By doing so, we could calculate a build plan on our monorepo for the workflow that should be run specifically for what projects in the monorepo have changed on the branch. This would make it easier for us to run jobs in parallel and kick off a build flow that's much more dynamic.

  • Avatar32.5fb70cce7410889e661286fd7f1897de Guest
  • Jan 29 2018
  • Taking votes
  • Attach files
  • Admin
    Nathan Dintenfass commented
    April 10, 2018 19:35

    This kind of functionality is on our radar. Between now and then we are likely to offer other forms of dynamism that are closer to our current config paradigm (allowing for parameterized/shared bits of config as well as looking at possibly allowing filtering based on where files have changed to accommodate monorepos that want to run conditional workflows/jobs based on what parts of the repo are being changed). Along with that work will come a more formal schema for a fully-expanded form of our config that will be needed as a target for any dynamic config creation.

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    25 Feb 16:51

    +1

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    26 Feb 07:35

    +1

  • Admin
    Nathan Dintenfass commented
    15 May 13:44

    Pasting in details from a duplicate request (https://ideas.circleci.com/ideas/CCI-I-249):

    ```As a developer, I want the ability to generate my pipeline at runtime so that I can customize the stages and steps before execution. The idea being to "run this thing to generate the configuration (or config.yml) instead of hardcoding everything".

     A few reasons I think this is useful:

    1) Semi dynamic pipelines - not fully dynamic like Jenkins Pipelines, but more flexibility compared to now with what I would say is not much complexity for users

    2) Less templating - IMO templating in YAML files is confusing. Thanksfully, there isn't much now but I would be dissapointed.

    3) No need to invent additional YAML features. GitLab CI recently introduced the include directive, which I think is a massive mistake.

    4) One of my use cases is to build against multiple different platforms with fanout and fan in when those versions are not hardcoded into the YAML file.```

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    15 May 15:06

    Could you give an update for where this stands currently, or maybe even ETA?

  • Admin
    Nathan Dintenfass commented
    15 May 15:32

    @Ulrich Unfortunately, I don't have a date I can share. 

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    13 Aug 20:45

    This is something that buildkite does really well - I hope you guys can implement something like this soon, or at least give us the ability to specify conditional branches of a pipeline when doing monorepo builds.