Feature Requests

Tell us what you'd like to see added to CircleCI.
Add ZST compression as optional compression method to persist_to_workspace
The compression and decompression step of persist_to_workspace takes an very long time, e.g. when using docker save to save a couple of images to the workspace.Adding ZST compression can dramatically speed up this step for very similar compression ratios.See https://engineering.fb.com/core-data/zstandard/ and https://github.com/facebook/zstdTesting zstd vs tar locally for the result of images saved via docker save of file size 2.4GB.time zstd test.tar -o test.tar.zstzstd test.tar -o test.tar.zst 10.85s user 0.95s system 59% cpu 19.790 totaltime tar cvzf test.tar.gz test.tartar cvzf test.tar.gz test.tar 98.86s user 4.98s system 102% cpu 1:41.71 totalFile sizes:790M ./test.tar.gz730M ./test.tar.zstAnd using more optimal settings for ZST on a quad core machine:time zstd --long -T4 -f test.tar -o test.tar.zsttest.tar : 25.75% (2483335168 => 639374809 bytes, test.tar.zst) zstd --long -T4 -f test.tar -o test.tar.zst 20.75s user 0.87s system 189% cpu 11.390 total610M ./test.tar.zstThe performance gains are substantial an would place CircleCi workflow performance at a great advantage.Decompression speed:time zstd -d test.tar.zst -o test.zst-decomp zstd -d test.tar.zst -o test.zst-decomp 2.20s user 1.42s system 91% cpu 3.962 totaltime tar -xvzf test.tar.gz tar -xvzf test.tar.gz 15.81s user 5.45s system 104% cpu 20.313 totalSome versions of tar support zst compression: https://www.gnu.org/software/tar/manual/html_node/gzip.html CCI-I-1659
0
Load More