When building locally using circleci CLI, it should be possible to ignore particular files. This would allow to ignore files with permissions that prevent the user inside docker from reading a file.
This may be achieved by using an existing .dockerignore file (as in my case it is the same files that need to be excluded for docker builds with and without circleci). Or it could be achieved by adding a circleci specific ignore file.