CircleCI Ideas

Local orbs in seperate yml files.

Orbs are great as reusable components but also considerably tidy up individual components as part of single, complicated build setups.

Currently there is a way to do local, "private" orbs "locally" by including the orb configuration under a key in `.circleci/config,yml`. However this still means we must maintain a giant singular yml file which can grow to thousands of lines potentially with all the workflows etc.

A "simple" solution for local orbs would be if circleci read additional .yml files in `.circleci/orbs/*` for a given project and merged the contents under the `orbs` key.

 

This effectively means that the yml contents of `.circleci/orbs/hello.yml` would be equivalent to defining the same content under 

 

orbs:
hello:

This allows circleci build components to be worked on in modular fashion and provides a step between a local orb that is "inline" and one that must be maintained completely externally from the project.

  • Avatar32.5fb70cce7410889e661286fd7f1897de Guest
  • Nov 26 2018
  • Taking votes
  • Attach files
  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    November 28, 2018 22:10

    To take it a step further, I think the `.circle/config.yml` file should always be processed with `circleci config pack` which would allow us to even split large workflows and jobs into their own files. 

     

    /2cents

  • Admin
    Nathan Dintenfass commented
    November 29, 2018 17:49

    We agree this would be useful, and we are working through approaches here. One of the reasons we didn't do this originally was that pulling lots of files before your jobs run risks hitting API rate limiting problems with GitHub (et al.) because we'd be pulling an arbitrary set of files via the API. That said, our current thinking is to create a mechanism where you can execute a job that can dynamically decide what the configuration should be -- with that capability doing `circleci config pack` to conjure your configuration from a decomposed orb (or set of orbs) will become fairly trivial to do yourself.

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    26 Apr 14:44

    Another option to consider: allow the "orbs" section to specify a reference to a URL that the orbs can be read from (so that private orbs could be hosted in S3, for example), or to a Docker image (and a path to a file within that image) that the orbs can be read from.  This would make it easy for organizations to host private orb repositories, without running into the API rate limit you mentioned.  It would also allow for easy sharing of orbs across private organization repositories.

    We'd really like to be able to maintain a set of common orbs across all of our projects, but for various compliance reasons we can't just put them where they are publicly visible, so hosting them in the public repository isn't an option for us.