CircleCI Ideas

Monorepo Support and/or filtering based on VCS changes in a specific directory

Currently, any merges to master trigger all steps of all workflows. In an monorepo that houses multiple projects, this results in multiple redundant steps as, for example, the backend is re-tested and re-deployed even if only frontend changes have been made.

 

It is currently possible to do:

 

workflows:
flow:
jobs:
- deploy:
filters:
branches:
only:
- master

 

It would be useful for all teams that run a monorepo to also be able to specify, say:

 

workflows:
flow:
jobs:
- terraform:
filters:
modified_paths:
only:
- /^infrastructure/
- deploy:
requires: terraform
filters:
modified_paths:
only:
- /^app/
  • Avatar32.5fb70cce7410889e661286fd7f1897de Guest
  • Feb 4 2018
  • Candidate
  • Attach files
  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    30 Jun 06:15am

    Any updates?

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    24 May 02:41am

    This is the best solution I can find. Any update on this?

    https://github.com/labs42io/circleci-monorepo

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    24 Apr 06:19am

    This is THE most requested feature and there hasn't been any progress in a year.

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    18 Apr 06:24pm
    We are definitely aware our monorepo support can be improved in various ways. A short-term fix will be to better surface information about the delta each trigger represents in terms of git SHAs. Longer-term, our plans are to allow more dynamic creation of config through inspection of the state of code. The idea is to give you power to do arbitrary work to then decide what the shape of your workflows should be. That work is currently sitting behind some other refactoring going on and finishing our roll out of pipelines for all projects.

    It's been a year since this. What's the update?

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    14 Apr 03:00am
  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    1 Apr 02:57pm

    I will most likely be moving at least some of our projects off of circleci and to github actions if this is not implemented or at least a timeline shared in the next few months. Definitely needed.

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    30 Mar 07:28pm

    This is a very useful feature for event-based micro-service architectures where a mono-repo is used to house directories of code based on business logic domain.

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    28 Jan 08:11pm

    Y'all should try out Buildkite in the meantime, which does support dynamic pipeline generation.

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    14 Oct, 2019 05:00pm

    I needed this and so I came up with an orb that has "run_if_modified" you can run only folders that are modified. Not ideal, but the best we can do now?

    https://circleci.com/orbs/registry/orb/roopakv/swissknife

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    9 Oct, 2019 07:02pm

    There is an article that describes how to setup monorepo with CircleCI conditional workflows and API v2. A working example can be found in this github repository.

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    20 Sep, 2019 07:02pm

    Monorepos support is a really needed feature, especially given the lack of ability to reuse code across multiple repositories (as orbs are at the time of writing still a public only thing which is not good enough for most companies). Moving multiple repos into a monorepo means more code reuse of similar workflows for similar projects is possible, provided circleci can support it.

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    27 Aug, 2019 04:14pm

    Heck yeah, that would be extremely helpful.

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    13 Aug, 2019 02:25am

    I'd also like this!

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    22 Jul, 2019 10:53pm
    Something like what you are proposing is under design -- we are working out a few details like what to do when receiving an API call directly to a branch rather than a webhook that has a commit -- the latter has a fairly clear set of changes, but the former is somewhat more ambiguous about what "changed" (should it be from previous build vs. previous commit, for instance). We do hope to work on this once we clear out our current backlog, though, but we can't yet predict timing of release.
    @Nathan Dintenfass it's been almost a year since you wrote this - any idea of timelines?
  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    24 Jun, 2019 08:32pm

    Unfortunately dnephin/multirepo doesn't match the use case here, we want the pipeline not to be run at all in this scenario.

    In addition, there are no examples for the orb and it seems to have a bug where it's run twice too. It also doesn't work if you need to use 'attach_at_workspace' in your command.

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    24 Jun, 2019 10:48am

    Some techniques for dealing with multiple projects in the same repo can be found in this orb: https://circleci.com/orbs/registry/orb/dnephin/multirepo

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    13 Jun, 2019 02:10pm

    Please please please. Without this it increases the complexity and time of our builds dramatically.

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    25 May, 2019 07:23pm

    Stretch goal: iOS and Android builds in one repo! 

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    25 May, 2019 07:23pm

    This would be extremely welcome. At the moment I don't know how we'd efficiently do this in Circle (maybe via Jenkins?), this would really help us out.

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    10 Apr, 2019 09:47pm

    We are definitely aware our monorepo support can be improved in various ways. A short-term fix will be to better surface information about the delta each trigger represents in terms of git SHAs. Longer-term, our plans are to allow more dynamic creation of config through inspection of the state of code. The idea is to give you power to do arbitrary work to then decide what the shape of your workflows should be. That work is currently sitting behind some other refactoring going on and finishing our roll out of pipelines for all projects.

  • Load older comments