bug: Zombie processes with host runner

fpedrei commented

2025-08-24 09:13:48 +00:00

Can you reproduce the bug on the Forgejo test instance?

No

Description

I'm running the official forgejo-runner container. Some actions use a self-hosted:host label and therefore run commands directly inside the container. When a running action is cancelled due to new commits, the runner stops the currently running action. This produces zombie processes and eventually blocks the runner.

It looks like the forgejo-runner process does not call waitpid(4) for its grandchildren here.

Forgejo Version

11

Runner Version

9

How are you running Forgejo?

Self hosted on a raspberry pi

How are you running the Runner?

Running as a podman container.

Logs

n/a

Workflow file

n/a

### Can you reproduce the bug on the Forgejo test instance? No ### Description I'm running the official forgejo-runner container. Some actions use a `self-hosted:host` label and therefore run commands directly inside the container. When a running action is cancelled due to new commits, the runner stops the currently running action. This produces zombie processes and eventually blocks the runner. It looks like the `forgejo-runner` process does not call `waitpid(4)` for its grandchildren here. ### Forgejo Version 11 ### Runner Version 9 ### How are you running Forgejo? Self hosted on a raspberry pi ### How are you running the Runner? Running as a podman container. ### Logs n/a ### Workflow file n/a

👍 1

fpedrei added the

Kind/Bug

label

2025-08-24 09:13:48 +00:00

earl-warren commented

2025-08-24 09:45:18 +00:00

Contributor

The runner does not set a process group where it should, I think. It should use something similar to https://code.forgejo.org/f3/gof3/src/branch/main/util/exec.go and kill the entire process group to ensure there is no leftover. Of course it can't help the case where a sub-process establishes its own process group and/or runs something in the background.

Since there is no way to guarantee that jobs are 100% sure to be cleaned up (such a guarantee requires containers), I'm inclined to say this is not in scope for the runner. However, it certainly deserves to be documented.

Does that make sense? Or do you think there is a way for the runner to provide guarantees to cleanup everything when running jobs directly on the host?

The runner does not set a process group where it should, I think. It should use something similar to https://code.forgejo.org/f3/gof3/src/branch/main/util/exec.go and kill the entire process group to ensure there is no leftover. Of course it can't help the case where a sub-process establishes its own process group and/or runs something in the background. Since there is no way to guarantee that jobs are 100% sure to be cleaned up (such a guarantee requires containers), I'm inclined to say this is not in scope for the runner. However, it certainly deserves to be documented. Does that make sense? Or do you think there is a way for the runner to provide guarantees to cleanup everything when running jobs directly on the host?

earl-warren commented

2025-08-24 09:53:20 +00:00

Contributor

Proposed an update to the documentation at https://codeberg.org/forgejo/docs/pulls/1406

earl-warren commented

2025-08-27 07:47:18 +00:00

Contributor

Closing as resolved by https://codeberg.org/forgejo/docs/pulls/1406

earl-warren

2025-08-27 07:47:19 +00:00

closed this issue
added the
Kind/Documentation
label

fpedrei commented

2025-08-29 06:29:52 +00:00

Author

Since there is no way to guarantee that jobs are 100% sure to be cleaned up

That's hillarious, cleaning leftover processes status does not require container technology. /sbin/init could do this decades ago. :-)

I think, that you didn't get the point. It is not the problem, that the grandchild processes do not terminate. In fact they do and they leave zombie processes lying around. It is forgejo-runners duty to collect their exit status with waitpid(2) until no result code is returned. Otherwise these zombies clutter the system.

If forgejo-runner should scan for running processes started by actions and kill them, is a different topic. Although I would vote for it.

Please re-open this issue and leave it open until a fix was provided. There are users forging software on *nix system without the fairy dust of container technology.

> Since there is no way to guarantee that jobs are 100% sure to be cleaned up That's hillarious, cleaning leftover processes status does not require container technology. `/sbin/init` could do this decades ago. :-) I think, that you didn't get the point. It is not the problem, that the grandchild processes do not terminate. In fact they do and they leave zombie processes lying around. It is forgejo-runners duty to collect their exit status with `waitpid(2)` until no result code is returned. Otherwise these zombies clutter the system. If `forgejo-runner` should scan for running processes started by actions and kill them, is a different topic. Although I would vote for it. Please re-open this issue and leave it open until a fix was provided. There are users forging software on *nix system without the fairy dust of container technology.

fpedrei reopened this issue

2025-08-29 06:29:52 +00:00

earl-warren commented

2025-08-29 07:02:30 +00:00

Contributor

Please consider the following scenario:

Job runs and forks a daemon.
Job terminates.
The daemon keeps running in the background.

When a job runs out of a host, there is no way for the runner to know that this daemon was spawned by the job and needs cleanup.

@fpedrei wrote in #911 (comment):

That's hillarious, cleaning leftover processes status does not require container technology. /sbin/init could do this decades ago. :-)

This is not what I was referring to. Adding an init daemon to the container would do that, sure. But it is a bandaid rather than a solution.

Jobs that run test may not just leave lingering process, they are also susceptible of removing files from the host and make it unusable. It is also not possible for the runner to ensure isolation on that level when containers are not available.

A personal note on the tone of your message:

That's hillarious,...
There are users forging software on *nix system without the fairy dust of container technology.

It reads like you are mocking my reply and look down on my technical skills. Even if you were right, this is unpleasant to read and I hope we can communicate in a more friendly tone in the future.

Please consider the following scenario: - Job runs and forks a daemon. - Job terminates. - The daemon keeps running in the background. When a job runs out of a host, there is no way for the runner to know that this daemon was spawned by the job and needs cleanup. @fpedrei wrote in https://code.forgejo.org/forgejo/runner/issues/911#issuecomment-57227: > That's hillarious, cleaning leftover processes status does not require container technology. `/sbin/init` could do this decades ago. :-) This is not what I was referring to. Adding an init daemon to the container would do that, sure. But it is a bandaid rather than a solution. Jobs that run test may not just leave lingering process, they are also susceptible of removing files from the host and make it unusable. It is also not possible for the runner to ensure isolation on that level when containers are not available. --- A personal note on the tone of your message: > That's hillarious,... > There are users forging software on *nix system without the fairy dust of container technology. It reads like you are mocking my reply and look down on my technical skills. Even if you were right, this is unpleasant to read and I hope we can communicate in a more friendly tone in the future.

fpedrei commented

2025-09-02 14:43:55 +00:00

Author

I think, that you didn't the point here. Let's examine the situation of the issue reported. This is not about running processes.

After some jobs, the runner looks like this:

# ps -ef --forest
UID          PID    PPID  C STIME TTY          TIME CMD
[...]
root     1211818       1  0 21:29 ?        00:00:00 /usr/bin/conmon --api-version 1 -c 911[...]0c
forgejo    1211820 1211818  0 21:29 pts/0    00:00:03  \_ forgejo-runner daemon --config config.yml
forgejo    1211937 1211820  0 21:29 ?        00:00:00      \_ [git] <defunct>
forgejo    1221082 1211820  0 21:39 ?        00:00:00      \_ [git] <defunct>

As you can see, all processes have been terminated. The ps(1) man page says:
"Processes marked are dead processes (so-called "zombies") that remain because their parent has not destroyed them properly. These processes will be destroyed by init(8) if the parent process exits."

This is worked out in waitpid(2) man page a little bit:

"In the case of a terminated child, performing a wait allows the system to release the resources associated with the child; if a wait is not performed, then the terminated child remains in a "zombie" state [...].

A child that terminates, but has not been waited for becomes a "zombie". The kernel maintains a minimal set of information about the zombie process (PID, termination status, resource usage information) in order to allow the parent to later perform a wait to obtain information about the child. As long as a zombie is not removed from the system via a wait, it will consume a slot in the kernel process table, and if this table fills, it will not be possible to create further processes. If a parent process terminates, then its "zombie" children (if any) are adopted by init(1),"

A look inside forgejo-runner shows, that the responsible process (with PID 1) for collecting orphaned exit codes is forgejo-runner itself.

# podman exec -it forgejo-runner-5 ps w
PID   USER     TIME  COMMAND
    1 1000      0:03 forgejo-runner daemon --config config.yml
   91 1000      0:00 [git]
  288 1000      0:00 [git]
 5938 1000      0:00 ps w

As you've suggested a possible solution would be a dedicated init process here. This could be something like adding tini to the container, or starting the container with --init. Another solution would let forgejo-runner collect outstanding exit codes.

I suppose, that you agree, that using host based runners might have security implications. But if you're accepting these and know, that your CI processes do no harm, they should get cleaned up, once they terminate.

I think, that you didn't the point here. Let's examine the situation of the issue reported. This is not about _running_ processes. After some jobs, the runner looks like this: ``` # ps -ef --forest UID PID PPID C STIME TTY TIME CMD [...] root 1211818 1 0 21:29 ? 00:00:00 /usr/bin/conmon --api-version 1 -c 911[...]0c forgejo 1211820 1211818 0 21:29 pts/0 00:00:03 \_ forgejo-runner daemon --config config.yml forgejo 1211937 1211820 0 21:29 ? 00:00:00 \_ [git] <defunct> forgejo 1221082 1211820 0 21:39 ? 00:00:00 \_ [git] <defunct> ``` As you can see, all processes have been terminated. The `ps(1)` man page says: "Processes marked <defunct> are dead processes (so-called "zombies") that remain because their parent has not destroyed them properly. These processes will be destroyed by init(8) if the parent process exits." This is worked out in `waitpid(2)` man page a little bit: "In the case of a terminated child, performing a wait allows the system to release the resources associated with the child; if a wait is not performed, then the terminated child remains in a "zombie" state [...]. A child that terminates, but has not been waited for becomes a "zombie". The kernel maintains a minimal set of information about the zombie process (PID, termination status, resource usage information) in order to allow the parent to later perform a wait to obtain information about the child. As long as a zombie is not removed from the system via a wait, it will consume a slot in the kernel process table, and if this table fills, it will not be possible to create further processes. If a parent process terminates, then its "zombie" children (if any) are adopted by init(1)," A look inside `forgejo-runner` shows, that the responsible process (with PID 1) for collecting orphaned exit codes is `forgejo-runner` itself. ``` # podman exec -it forgejo-runner-5 ps w PID USER TIME COMMAND 1 1000 0:03 forgejo-runner daemon --config config.yml 91 1000 0:00 [git] 288 1000 0:00 [git] 5938 1000 0:00 ps w ``` As you've suggested a possible solution would be a dedicated init process here. This could be something like adding `tini` to the container, or starting the container with `--init`. Another solution would let `forgejo-runner` collect outstanding exit codes. I suppose, that you agree, that using host based runners might have security implications. But if you're accepting these and know, that your CI processes do no harm, they should get cleaned up, once they terminate.

earl-warren commented

2025-09-02 15:19:45 +00:00

Contributor

@fpedrei wrote in #911 (comment):

I think, that you didn't the point here.

Is there a word missing in this sentence?

@fpedrei wrote in https://code.forgejo.org/forgejo/runner/issues/911#issuecomment-57698: > I think, that you didn't the point here. Is there a word missing in this sentence?

earl-warren commented

2025-09-02 15:22:54 +00:00

Contributor

@fpedrei wrote in #911 (comment):

As you've suggested a possible solution would be a dedicated init process here. This could be something like adding tini to the container, or starting the container with --init. Another solution would let forgejo-runner collect outstanding exit codes.

I agree that it would help solve that particular problem. It would not help for daemons running in the background and other various leftovers that a faulty test can create, but it would take care of that.

Since this is an improvement over the current implementation rather than a bug by which it does not work as intended, please file a feature request.

@fpedrei wrote in https://code.forgejo.org/forgejo/runner/issues/911#issuecomment-57698: > As you've suggested a possible solution would be a dedicated init process here. This could be something like adding `tini` to the container, or starting the container with `--init`. Another solution would let `forgejo-runner` collect outstanding exit codes. I agree that it would help solve that particular problem. It would not help for daemons running in the background and other various leftovers that a faulty test can create, but it would take care of that. Since this is an improvement over the current implementation rather than a bug by which it does not work as intended, please file [a feature request](https://code.forgejo.org/forgejo/forgejo-actions-feature-requests/issues/new?template=.forgejo%2fissue_template%2ffeature-request.yaml).

earl-warren closed this issue

2025-09-08 10:24:57 +00:00

fpedrei commented

2025-09-15 16:53:23 +00:00

Author

Good that you want to improve here.

Please keep this issue (orwhatever you call it) open until it is fixed.

Good that you want to improve here. Please keep this issue (orwhatever you call it) open until it is fixed.

fpedrei reopened this issue

2025-09-15 16:53:27 +00:00

earl-warren commented

2025-09-16 06:52:44 +00:00

Contributor

I created a feature request at forgejo/forgejo-actions-feature-requests#50, the discussion continues there.

I created a feature request at https://code.forgejo.org/forgejo/forgejo-actions-feature-requests/issues/50, the discussion continues there.

earl-warren closed this issue

2025-09-16 06:52:44 +00:00

Rows
Columns

bug: Zombie processes with host runner #911

Can you reproduce the bug on the Forgejo test instance?

Description

Forgejo Version

Runner Version

How are you running Forgejo?

How are you running the Runner?

Logs

Workflow file