CircleCI Ideas

Allow orbs to include a non-YAML directory tree payload

With Docker I can easily pack up all the scripts I need for my orbs into the image and get at this easily at runtime. But for all other build types this is much more difficult. It would be fantastic if orbs could include an arbitrary payload of files that would be unpacked in a known location like "~/orbs/$ORB_NAME/$ORIGINAL_TREE".

  • Avatar32.5fb70cce7410889e661286fd7f1897de Guest
  • Oct 19 2019
  • New
  • Attach files
  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    October 19, 2019 02:06

    I did find a hilarious workaround.

     

    I made a tarball of the directory tree, then I ran it through base64 to encode it. Then as one of my macOS build steps I have this:

     command: |
    echo 'H4sIAAdqql0AA+w9/VfbuLL7c/4KbZq9Tt5pvkgC94QLtxRoy7nQ8oB29962m3UcJXFxbNd2oLRL
    [snip many lines] ...
    yAAA' | base64 --decode > /tmp/tools.tar.gz
    mkdir ~/circleci-perl-helpers-tools
    tar -xvzf /tmp/tools.tar.gz --strip-components 2 --directory ~/circleci-perl-helpers-tools
     
  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    October 19, 2019 02:08

    How would you want to represent that in the orb, itself? One option for this is there's some way to encode the files and insert them during configuration "packing" (a feature of our CLI that implements FYAML - https://github.com/CircleCI-Public/fyaml). This would make the actual orb source quite large and wouldn't be practical unless you had a publishing pipeline that uses config packing via the CLI (something many people do today). You could "fake" this fairly easily if you have a scripted orb publishing pipeline by encoding the files and generating a "big" shell command that uses the encoded string in a script. A more radical approach here would be to introduce a way to register files in a namespace and let an orb reference files at a given version as a first-class object. Files, unlike orb code, are often much more likely to contain proprietary information or secrets, so there's a strong argument for not encouraging them as part of configuration code and treat them more like secrets.

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    October 19, 2019 02:09

    (nice to see we arrived at the same workaround - literally while I was typing)

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    October 19, 2019 04:20

    @Nathan - I'm suggesting that an orb should just be a tarball that contains a single config.yml file and any number of other files. CircleCI could then pull out the config.yml and just untar the rest of it into the home directory of the image/VM where it runs the CI job.

    I think if people want to publish files with secret info that's a different problem entirely, one that would probably be better handled by supporting some sort of private orb publishing process for paying customers.