Docker/Podman inside unprivileged podman

jornfranke commented

2023-12-24 18:33:57 +00:00

Hallo,

thanks a lot for providing this! I managed successfully to run a Codeberg Action pipeline using a forgejo-runner deployed inside an unpriviledged podman in an IPv6 only network:
https://jornfranke.codeberg.page/technology-tutorials/forgejo-runner-opensuse-microos-hetzner/

See here: https://codeberg.org/jornfranke/forgejo-runner-test/actions/runs/4

Now, I want to go a step further and allow in the Codeberg action a custom container image to load a specific toolchain (Java, Rust etc.), ie the forgejo-runner inside the unpriviledged podman needs to spawn a new container. As far as I understood, forgejo-runner uses docker for this (btw. any chance one can contribute a podman variant)?

So what I did is - as the unpriviledged user I create a podman.sock:

podman system service -t 0 &

Then I run the forgejo runner (I add here only the relevant command):

podman run --security-opt label=disable -e DOCKER_HOST=unix:///var/run/podman/podman.sock -v /run/user/$(id --user)/podman/:/var/run/podman/:Z -v /var/tmp/${RUNNER_NAME}/data:/data:Z -v /home/${RUNNER_NAME}/runnerconf:/home/runner:Z --rm code.forgejo.org/forgejo/runner:\${RUNNER_VERSION} forgejo-runner daemon -c /home/runner/config.yml

Now I try to run the following Action pipeline: 6ac32c8cb2

It seems initially go well - Docker inside podman fetches the container image: https://codeberg.org/jornfranke/forgejo-runner-test/actions/runs/11

This means Docker can successfully connect to the unprivileded podman.sock (Before I had the wrong permissions which showed another error before pulling the image).

However, as you can see it fails with the message

failed to create container: 'Error response from daemon: make cli opts(): making volume mountpoint for volume /var/run/podman/podman.sock: mkdir /var/run/podman/podman.sock: permission denied'

It seems when starting the image, it tries to override/mount the podman.sock? Can you explain how forgejo-runner initiates Docker and or how I can fix this?

Will add this to my public instructions mentioned above.

Hallo, thanks a lot for providing this! I managed successfully to run a Codeberg Action pipeline using a forgejo-runner deployed inside an unpriviledged podman in an IPv6 only network: https://jornfranke.codeberg.page/technology-tutorials/forgejo-runner-opensuse-microos-hetzner/ See here: https://codeberg.org/jornfranke/forgejo-runner-test/actions/runs/4 Now, I want to go a step further and allow in the Codeberg action a custom container image to load a specific toolchain (Java, Rust etc.), ie the forgejo-runner inside the unpriviledged podman needs to spawn a new container. As far as I understood, forgejo-runner uses docker for this (btw. any chance one can contribute a podman variant)? So what I did is - as the unpriviledged user I create a podman.sock: ``` podman system service -t 0 & ``` Then I run the forgejo runner (I add here only the relevant command): ``` podman run --security-opt label=disable -e DOCKER_HOST=unix:///var/run/podman/podman.sock -v /run/user/$(id --user)/podman/:/var/run/podman/:Z -v /var/tmp/${RUNNER_NAME}/data:/data:Z -v /home/${RUNNER_NAME}/runnerconf:/home/runner:Z --rm code.forgejo.org/forgejo/runner:\${RUNNER_VERSION} forgejo-runner daemon -c /home/runner/config.yml ``` Now I try to run the following Action pipeline: https://codeberg.org/jornfranke/forgejo-runner-test/commit/6ac32c8cb2f2873784a6d50c627a9b4c0909ce38 It seems initially go well - Docker inside podman fetches the container image: https://codeberg.org/jornfranke/forgejo-runner-test/actions/runs/11 This means Docker can successfully connect to the unprivileded podman.sock (Before I had the wrong permissions which showed another error before pulling the image). However, as you can see it fails with the message ``` failed to create container: 'Error response from daemon: make cli opts(): making volume mountpoint for volume /var/run/podman/podman.sock: mkdir /var/run/podman/podman.sock: permission denied' ``` It seems when starting the image, it tries to override/mount the podman.sock? Can you explain how forgejo-runner initiates Docker and or how I can fix this? Will add this to my public instructions mentioned above.

👀 1

jornfranke commented

2023-12-25 21:16:52 +00:00

Author

I seem to got around it.

Solution is in: https://gitea.com/gitea/act_runner/issues/223:

Update the config.yml and include the following

container:
     docker_host: "-"

still tuning a bit the options, but I can run the forgejo-runner in rootless podman.

Full command:

podman run --security-opt label=disable -e DOCKER_HOST=unix:///var/run/docker.sock -v /run/user/$(id --user)/podman/podman.sock:/var/run/docker.sock:Z -v /var/tmp/${RUNNER_NAME}/data:/data:Z -v /home/${RUNNER_NAME}/runnerconf:/home/runner:Z --rm code.forgejo.org/forgejo/runner:\${RUNNER_VERSION} forgejo-runner daemon -c /home/runner/config.yml

Since the forgejo-runner protocol is open and if I have some time (...) I wonder if I can do something also with firecracker (https://github.com/firecracker-microvm/firecracker) - this would isolate it a bit more and gives also some new possibilities to play with (end in the end I could use also an alternative to Forgejo Actions to define the pipeline if I wanted, e.g. https://taskfile.dev/). For sure interesting approach for the Forgejo runner that it can be so flexible.

I seem to got around it. Solution is in: https://gitea.com/gitea/act_runner/issues/223: Update the config.yml and include the following ``` container: docker_host: "-" ``` still tuning a bit the options, but I can run the forgejo-runner in rootless podman. Full command: ``` podman run --security-opt label=disable -e DOCKER_HOST=unix:///var/run/docker.sock -v /run/user/$(id --user)/podman/podman.sock:/var/run/docker.sock:Z -v /var/tmp/${RUNNER_NAME}/data:/data:Z -v /home/${RUNNER_NAME}/runnerconf:/home/runner:Z --rm code.forgejo.org/forgejo/runner:\${RUNNER_VERSION} forgejo-runner daemon -c /home/runner/config.yml ``` Since the forgejo-runner protocol is open and if I have some time (...) I wonder if I can do something also with firecracker (https://github.com/firecracker-microvm/firecracker) - this would isolate it a bit more and gives also some new possibilities to play with (end in the end I could use also an alternative to Forgejo Actions to define the pipeline if I wanted, e.g. https://taskfile.dev/). For sure interesting approach for the Forgejo runner that it can be so flexible.

👍 1

jornfranke closed this issue

2023-12-25 21:16:52 +00:00

earl-warren commented

2023-12-25 21:27:06 +00:00

Owner

Excellent update, will be most useful to people looking for answers 🙏

jornfranke commented

2023-12-26 22:23:17 +00:00

Author

Finally, I could also a container image in buildah (non-privileged and rootless) :) (just needed to provide --device /dev/fuse to the container ...). Instructions under the same link
Also activated cache of forgejo-runner (a suggestion here could be to use simply a volume instead of a http url...).
Thanks again.

Finally, I could also a container image in buildah (non-privileged and rootless) :) (just needed to provide --device /dev/fuse to the container ...). Instructions under the same link Also activated cache of forgejo-runner (a suggestion here could be to use simply a volume instead of a http url...). Thanks again.

Xinayder commented

2023-12-27 09:13:31 +00:00

Member

I wonder if this can be applied to the Docker-in-Docker example for a runner inside a k8s cluster. AFAIK the current example is running in a privileged container: https://code.forgejo.org/forgejo/runner/src/branch/main/examples/kubernetes/dind-docker.yaml

jornfranke commented

2023-12-27 18:19:43 +00:00

Author

Probably not, I would generally advise against Docker (especially DinD which is even worse).
However, if you use a recent Kubernetes without Docker you could run the forgejo image in it and then follow the instructions, there would be no DinD needed.

Maybe (however this is optional and not needed to get rid of DinD), we could ship the "official" forgejo-runner image without docker and install there podman and the package podman-docker (this is available in many distributions and essentially provided symbolic links so that your docker cli commands work as expected in podman).

Podman is the official container tool in many Linux distributions. k8s anyway sunset Docker some time ago (https://kubernetes.io/blog/2020/12/02/dont-panic-kubernetes-and-docker/). Podman has been designed from the start to work in a rootless and non-priviledged scenario.

I will also do some further experiments to avoid needing to share /dev/fuse - lets see if I will be successful - this might be interesting for Kubernetes.

Nevertheless, Kubernetes itself is also not secure enough to run runners e.g. of different organisations / organisational units in the same cluster (independent what software, such as runners, you deploy there). Depends of course on the security posture of your organisation.

Probably not, I would generally advise against Docker (especially DinD which is even worse). However, if you use a recent Kubernetes without Docker you could run the forgejo image in it and then follow the instructions, there would be no DinD needed. Maybe (however this is optional and not needed to get rid of DinD), we could ship the "official" forgejo-runner image without docker and install there podman and the package podman-docker (this is available in many distributions and essentially provided symbolic links so that your docker cli commands work as expected in podman). Podman is the official container tool in many Linux distributions. k8s anyway sunset Docker some time ago (https://kubernetes.io/blog/2020/12/02/dont-panic-kubernetes-and-docker/). Podman has been designed from the start to work in a rootless and non-priviledged scenario. I will also do some further experiments to avoid needing to share /dev/fuse - lets see if I will be successful - this might be interesting for Kubernetes. Nevertheless, Kubernetes itself is also not secure enough to run runners e.g. of different organisations / organisational units in the same cluster (independent what software, such as runners, you deploy there). Depends of course on the security posture of your organisation.

jornfranke commented

2023-12-27 18:54:20 +00:00

Author

And I got rid of the need for /dev/fuse by using vfs :) See updated instructions.

Note: If you are not happy with the performance (you have to test, depends a bit how you build containers), the annex still describes how you can use fuse - with the disadvantage that you need to expose the /dev/fuse

And I got rid of the need for /dev/fuse by using vfs :) See updated instructions. Note: If you are not happy with the performance (you have to test, depends a bit how you build containers), the annex still describes how you can use fuse - with the disadvantage that you need to expose the /dev/fuse

jornfranke commented

2023-12-27 19:47:12 +00:00

Author

nb: if we replace the docker inside the forgejo-runner container with podman, we can also avoid leaking the podman.sock (it is currently non-privileged and rootless, but still), cf. https://www.redhat.com/sysadmin/podman-inside-container (scroll to the end).

👍 1

Xinayder commented

2023-12-29 17:26:56 +00:00

Member

Probably not, I would generally advise against Docker (especially DinD which is even worse).
However, if you use a recent Kubernetes without Docker you could run the forgejo image in it and then follow the instructions, there would be no DinD needed.

k3s is a lightweight kubernetes engine and it uses containerd for managing containers. The problem is that the k8s runner example installs a pod with Docker in it, which will be used to run the runner inside docker, running inside k8s.

I'm no kube expert but I'd be very interested in a runner that doesn't run privileged or with root access. Right now I kind of need it to have the runner run as privileged because the action to build my website and publish a docker image doesn't work if it's access is not elevated.

> Probably not, I would generally advise against Docker (especially DinD which is even worse). However, if you use a recent Kubernetes without Docker you could run the forgejo image in it and then follow the instructions, there would be no DinD needed. k3s is a lightweight kubernetes engine and it uses containerd for managing containers. The problem is that the k8s runner example installs a pod with Docker in it, which will be used to run the runner inside docker, running inside k8s. I'm no kube expert but I'd be very interested in a runner that doesn't run privileged or with root access. Right now I kind of need it to have the runner run as privileged because the action to build my website and publish a docker image doesn't work if it's access is not elevated.

jornfranke commented

2024-01-04 18:39:37 +00:00

Author

Another update: I have changed the instructions to use quadlet (native podman/systemd iintegration, see here and podman-systemd. This is more elegant, simpler, provides further auto-update capabilities, declarative configuration (.container, .volume, .network, .kube), support of k8s manifests etc.

It does not solve exactly your Kubernetes problem, but quadlet can reuse your k8s yml files as well.

Another update: I have changed the instructions to use quadlet (native podman/systemd iintegration, see [here](https://www.redhat.com/sysadmin/quadlet-podman) and [podman-systemd](https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html). This is more elegant, simpler, provides further auto-update capabilities, declarative configuration (.container, .volume, .network, .kube), support of k8s manifests etc. It does not solve exactly your Kubernetes problem, but quadlet can reuse your k8s yml files as well.

👍 1

anri referenced this issue

2024-04-14 16:26:24 +00:00

Support for podman #158

Docker/Podman inside unprivileged podman #139