Custom Machine Executor AMIs
In the same way customers can create their own custom Docker images for execution (https://circleci.com/docs/2.0/custom-images/), it would be a great feature to have custom AMIs for the machine executor. Just tell us what needs to be pre-installed on these AMIs to run on your system.
This is especially useful for:
Cutting down build time associated with installing specific dependencies on a clean box
Removing complexity associated with installing specific dependencies programmatically
Martijn The here is the gitlab support for this : https://docs.gitlab.com/runner/executors/custom_examples/libvirt.html
I would really prefer benefitting of something like aws support : using Packer instead of raw libvirt to create a predefined image, then post-install scripts IMHO is not necessary, everything can be in the predefined image, and clean up well to me it's Gitlab's service resposibility to clean things up properly out of the box
It’s really about installing required software dependencies on every run - it adds latency and burns bandwidth - in our case we need latest erlang / elixir
We need fully customized AMI for Cloud Service (or, at least, AMI based on your public images), because some SDKs are outdated and we have to update them every run for every job dynamically. It is hard to maintain and adds overhead to all our builds.
I support the idea. Nevertheless, I would argue this :
* If we want to use a custom Virtual Machine image, it is mostly I believe, because we want to use features we cannot have in a Docker container. Those features are either hardware specific, like GPU-based etc... (CI CD Tests for a blockchain ?) , or it's about using docker in docker, with docker volumes mapped to the host.
* I'll try very soon to use docker containers with docker volumes, inside a docker container, inside a Machine executor. It's it's possible.
* Last, and not least, maintaining a Virtual machine Library, is a fair amount of work:
* much more complex than a docker image library (which is already is an amount of work though)
* If Had to do that, I would like to automate this maintainance, and also have a CI CD process over that maintainance : I would test, any VM image that I would publish, for my CI CD System to use it.
* And I would want my VM images CI CD Pipelines to be Circle CI Pipelines. Which means I would want to run my pipeleins, "in test mode", cf. https://ideas.circleci.com/api-feature-requests/p/behavior-triggering-pipeline-of-circleci-project-not-set-up-to-start-building
All in all, I think the best way to implement that, is to have a PXELess build service inside the Circle CI Service itself, sowe can build VM images with Terraform, just like we build AMIs in AWS with their PXEless internal service. A way for AWS to secure custom VM image creation
Also useful for CI jobs that involve custom hardware that needs to be attached to the executor. Gitlab does offer great support for this.
This would help us absolutely enormously. I can't figure out how to vote, but here's my vote!