Monorepo Support and/or filtering based on VCS changes in a specific directory
complete
J
Jamie Talbot
Currently, any merges to master trigger all steps of all workflows. In an monorepo that houses multiple projects, this results in multiple redundant steps as, for example, the backend is re-tested and re-deployed even if only frontend changes have been made.
It is currently possible to do:
workflows: flow: jobs: - deploy: filters: branches: only: - master
It would be useful for all teams that run a monorepo to also be able to specify, say:
workflows: flow: jobs: - terraform: filters: modified_paths: only: - /^infrastructure/ - deploy: requires: terraform filters: modified_paths: only: - /^app/
CCI-I-208
B
Brandon Doyle
I have an orb that will do this automatically.
https://github.com/bjd2385/dynamic-continuation-orb
Salil Subbakrishna
complete
Dynamic config has been released along with an orb to make this process easier.
B
Bibin Chandran
Salil Subbakrishna: Where is the orb? any documentation or demo video
Liya Ai
Bibin Chandran: Hi Bibin - you can read about the feature & continuation orb: https://circleci.com/docs/2.0/dynamic-config/. Thanks!
B
Bibin Chandran
Liya Ai: Thank you
Salil Subbakrishna
Hi all, I wanted to provide an update here. This feature is currently in closed preview. I'll continue to update as we get closer to release.
J
James Dunne
Salil Subbakrishna: Any chance of a sneak preview, like with a beta program? We're desperate for this!
Daniel 'f0o' Preussker
James Dunne: Salil Subbakrishna:
Same here, if possible please allow sneak previews ;)
Liya Ai
Daniel 'f0o' Preussker James Dunne: Good news! We've now made set-up workflows an open preview: you can find out more info & how to get set-up: https://discuss.circleci.com/t/setup-workflows-open-preview/39516
Salil Subbakrishna
in progress
M
Matthew Morrissette Vance
Salil Subbakrishna: Super happy to see this in progress.
If I wanted to set the pipeline parameters based on which files have changed since a base branch (not just in the current push), would this be possible?
Given the above description of the feature, let's say I branch from "develop" to a "feature/foo" branch and my first commit makes changes to some files in the "frontend" directory.
I push and using this new feature, CircleCI sees the changes in my frontend directory and triggers the frontend build job. Build completes with an error and I realize there is an issue that require me to change the CircleCI config file (in .circleci/config.yml).
I make the required change and push. This time because there are no changes in the frontend files, the frontend build does not run.
I do not want to add ".circleci/config.yml" to my filter that causes a frontend build when it changes as that means I would have to trigger an entire build of the entire monorepo every time I change the config file.
What I really want is to trigger the frontend job whenever the list of changed files between my branch and the base branch (i.e. files I have changed) includes a specific directory.
Salil Subbakrishna
Matthew Morrissette Vance: The way to do this would be to take our orb and fork it. There is a part of the orb that will run a few operations to determine what has changed (which will be mostly git commands), you would just need to alter that part of the orb to determine the files the way you want and everything else should work. Over time, we may add some of these behaviors to our orb if there is high adoption of a fork.
Salil Subbakrishna
We are still working out the specifics of the syntax, but at a high level, it will look like the following:
* You will be able to provide a set of mappings between regexes for paths and pipeline variables.
* We will provide orbs/jobs that you will call to set these pipeline parameters based upon whether the push includes modified files that match that path.
* You can use those pipeline parameters to filter in your config anywhere you could use a pipeline parameter, such as a "when" statement in workflow definition.
F
François Wouts
Salil Subbakrishna: This sounds amazing.
I think an important follow-up would be to be able to split up
config.yml
. In a monorepo with 100+ projects, it will quickly end up being many thousands of lines long, especially if there is a large variety of projects (e.g. Android + iOS + React + various backend frameworks) meaning they can't share orbs easily.Salil Subbakrishna
François Wouts: We're also looking into this and we expect a similar approach, using an orb with a setup workflow to collect/concatenate multiple smaller config files.
Alex Jurkiewicz
Have you decided on what the config syntax will be? Can you share it? It would be good to look at your planned work and know if it will work for us.
Salil Subbakrishna
Hi everyone, wanted to provide an update here. We are working on a way to filter which workflows run based on which files are modified as part of a git push. This will use the dynamic config features we are working on to set pipeline parameters based on which files are modified and these pipeline parameters can be used in when/unless statements in the config. We expect this to be available in the first quarter of 2021.
Salil Subbakrishna
planned
M
Matthieu Adjogah
Something like the job below should work for monorepos: unit_tests: executor: terraform_image steps: - cache_restoration_for_job_to_job - run: name: Run unit test for the updated modules only command: | # We get the commit paths, and extract only the 2 first levels of each path for change in $(echo "$(git diff-tree --no-commit-id --name-only -r HEAD)" | cut -d/ -f 1-2); do # Then we check that a subfolder "examples/integration-test" exists under what we got if [ -d "${change}/examples/integration-test" ]; then terraform init "${change}/examples/integration-test" terraform validate "${change}/examples/integration-test" fi done
A
Abcs
Any updates please?
Load More
→