CircleCI Ideas

Monorepo Support and/or filtering based on VCS changes in a specific directory

Currently, any merges to master trigger all steps of all workflows. In an monorepo that houses multiple projects, this results in multiple redundant steps as, for example, the backend is re-tested and re-deployed even if only frontend changes have been made.

 

It is currently possible to do:

 

workflows:
flow:
jobs:
- deploy:
filters:
branches:
only:
- master

 

It would be useful for all teams that run a monorepo to also be able to specify, say:

 

workflows:
flow:
jobs:
- terraform:
filters:
modified_paths:
only:
- /^infrastructure/
- deploy:
requires: terraform
filters:
modified_paths:
only:
- /^app/
  • Avatar32.5fb70cce7410889e661286fd7f1897de Guest
  • Feb 4 2018
  • On roadmap
  • Attach files
  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    February 07, 2018 06:49

    This feature is need for a organisation which has monorepo consists of multiple micro services as folder to avoid multiple repository overhead. 

    I am in need of this in my case i have a repo which consists of 5 micro service any changes made on single micro service triggers build for whole system which is more inefficient in this case.

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    February 28, 2018 19:35

    This is also required for us, with a very large monorepo.

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    April 03, 2018 08:24

    I have a similar use case and this would indeed help me very much. I started working on a custom script to hack this feature, you can see the diff here:

    https://github.com/decidim/decidim/pull/2028

    But if this feature was built-in it'd be super-awesome!

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    July 16, 2018 12:37

    This would be a great feature, because we have on big mono-repo with a workflow containing of 37 jobs.
    We have added a custom bash script which checks which files changed within a commit (or diff from master).
    But still all jobs are being queued and we can only check within the job itself - which means that image download and restoring of the cache are run in any case (means severals second - up to minutes a worker is blocked with nonsense work)

    Maybe it would be even better to be able to set a script (bash command or similar) as filter. Depending on the exit code the job will be queued.

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    August 13, 2018 19:33

    This is definitely necessary, lots of people use monorepos, including well-established authorities like Babel, Google, etc.

  • Admin
    Nathan Dintenfass commented
    August 13, 2018 22:46

    Something like what you are proposing is under design -- we are working out a few details like what to do when receiving an API call directly to a branch rather than a webhook that has a commit -- the latter has a fairly clear set of changes, but the former is somewhat more ambiguous about what "changed" (should it be from previous build vs. previous commit, for instance). We do hope to work on this once we clear out our current backlog, though, but we can't yet predict timing of release.

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    August 20, 2018 09:37

    This is is something that we'd benefit from as well. We'd probably get it to work with only regex on paths but my hunch is that it would be useful to be able to have a pre-process script that would deliver a list of jobs that should run (or something like that). We currently have at least jobs that should run if either path A or path B have changed.

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    August 25, 2018 22:43

    Yarrrrr! This would be lovely! Please oh please!

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    September 01, 2018 00:06

    This would be a truly needed feature... We're just about to start migrating to CircleCI - and this would save us...

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    September 29, 2018 18:24

    Just catching up on the features here and am pleased to see this is planned. Is there any estimate on the release of this feature?

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    November 14, 2018 14:30

    An absolute must in a modern JS monorepo setup.... upvote :)

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    November 26, 2018 21:37

    How far out is this planned feature?  I'm evaluating tools for our Monorepo and trying to see how this would impact things.

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    15 Jan 18:03

    Is there any update on the availability of this?

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    07 Feb 20:32

    any workaround on this one?

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    09 Mar 09:42

    Pretty essential for any team that works with mono repos, which seems to be on the rise lately.   Assuming 1 microservice per git repo is quite a big assumption for a CI tool....

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    01 Apr 14:09

    Please give us some information about a release date for this feature request ;)

  • Admin
    Nathan Dintenfass commented
    10 Apr 21:47

    We are definitely aware our monorepo support can be improved in various ways. A short-term fix will be to better surface information about the delta each trigger represents in terms of git SHAs. Longer-term, our plans are to allow more dynamic creation of config through inspection of the state of code. The idea is to give you power to do arbitrary work to then decide what the shape of your workflows should be. That work is currently sitting behind some other refactoring going on and finishing our roll out of pipelines for all projects.

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    25 May 19:23

    This would be extremely welcome. At the moment I don't know how we'd efficiently do this in Circle (maybe via Jenkins?), this would really help us out.

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    25 May 19:23

    Stretch goal: iOS and Android builds in one repo! 

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    13 Jun 14:10

    Please please please. Without this it increases the complexity and time of our builds dramatically.

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    24 Jun 10:48

    Some techniques for dealing with multiple projects in the same repo can be found in this orb: https://circleci.com/orbs/registry/orb/dnephin/multirepo

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    24 Jun 20:32

    Unfortunately dnephin/multirepo doesn't match the use case here, we want the pipeline not to be run at all in this scenario.

    In addition, there are no examples for the orb and it seems to have a bug where it's run twice too. It also doesn't work if you need to use 'attach_at_workspace' in your command.

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    22 Jul 22:53
    Something like what you are proposing is under design -- we are working out a few details like what to do when receiving an API call directly to a branch rather than a webhook that has a commit -- the latter has a fairly clear set of changes, but the former is somewhat more ambiguous about what "changed" (should it be from previous build vs. previous commit, for instance). We do hope to work on this once we clear out our current backlog, though, but we can't yet predict timing of release.
    @Nathan Dintenfass it's been almost a year since you wrote this - any idea of timelines?
  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    13 Aug 02:25

    I'd also like this!

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    27 Aug 16:14

    Heck yeah, that would be extremely helpful.