bug: failed to copy content to container (transient test error) #848

Closed
opened 2025-08-11 21:34:14 +00:00 by earl-warren · 5 comments
Contributor

https://code.forgejo.org/forgejo/runner/actions/runs/6639/jobs/1

2025-08-11T21:24:18.2243041Z --- FAIL: TestRunner_RunEvent (541.91s)
2025-08-11T21:24:18.2243129Z     --- FAIL: TestRunner_RunEvent/evalmatrix-merge-array (2.85s)
2025-08-11T21:24:18.2243223Z         runner_test.go:211: 
2025-08-11T21:24:18.2243318Z             	Error Trace:	/home/debian/.cache/act/f5d10de4023dec79/hostexecutor/act/runner/runner_test.go:211
2025-08-11T21:24:18.2243416Z             	            				/home/debian/.cache/act/f5d10de4023dec79/hostexecutor/act/runner/runner_test.go:362
2025-08-11T21:24:18.2243507Z             	Error:      	Received unexpected error:
2025-08-11T21:24:18.2243596Z             	            	copyContent: failed to copy content to container: Error response from daemon: Could not find the file /var/run/act/ in container e70a62fbc7663807b3776982af1c30eb362f082a35076ba82910f3b726b4985f
2025-08-11T21:24:18.2243709Z             	Test:       	TestRunner_RunEvent/evalmatrix-merge-array
2025-08-11T21:24:18.2243803Z             	Messages:   	/home/debian/.cache/act/f5d10de4023dec79/hostexecutor/act/runner/testdata/evalmatrix-merge-array
https://code.forgejo.org/forgejo/runner/actions/runs/6639/jobs/1 ``` 2025-08-11T21:24:18.2243041Z --- FAIL: TestRunner_RunEvent (541.91s) 2025-08-11T21:24:18.2243129Z --- FAIL: TestRunner_RunEvent/evalmatrix-merge-array (2.85s) 2025-08-11T21:24:18.2243223Z runner_test.go:211: 2025-08-11T21:24:18.2243318Z Error Trace: /home/debian/.cache/act/f5d10de4023dec79/hostexecutor/act/runner/runner_test.go:211 2025-08-11T21:24:18.2243416Z /home/debian/.cache/act/f5d10de4023dec79/hostexecutor/act/runner/runner_test.go:362 2025-08-11T21:24:18.2243507Z Error: Received unexpected error: 2025-08-11T21:24:18.2243596Z copyContent: failed to copy content to container: Error response from daemon: Could not find the file /var/run/act/ in container e70a62fbc7663807b3776982af1c30eb362f082a35076ba82910f3b726b4985f 2025-08-11T21:24:18.2243709Z Test: TestRunner_RunEvent/evalmatrix-merge-array 2025-08-11T21:24:18.2243803Z Messages: /home/debian/.cache/act/f5d10de4023dec79/hostexecutor/act/runner/testdata/evalmatrix-merge-array ```
Author
Contributor
2025-08-11T08:02:31.8870350Z [container-hostname/default-hostname] [DEBUG] Common container.HostConfig ==> &{Binds:[/var/run/docker.sock:/var/run/docker.sock] ContainerIDFile: LogConfig:{Type: Config:map[]} NetworkMode:WORKFLOW-0e329888b1e68284519f1ef64a178c87d53d4736a250a86f58a7b733f7249ba4_JOB-default-hostname-default-hostname-network PortBindings:map[] RestartPolicy:{Name: MaximumRetryCount:0} AutoRemove:true VolumeDriver: VolumesFrom:[] ConsoleSize:[0 0] Annotations:map[] CapAdd:[] CapDrop:[] CgroupnsMode: DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[] GroupAdd:[] IpcMode: Cgroup: Links:[] OomScoreAdj:0 PidMode: Privileged:false PublishAllPorts:false ReadonlyRootfs:false SecurityOpt:[] StorageOpt:map[] Tmpfs:map[] UTSMode: UsernsMode: ShmSize:0 Sysctls:map[] Runtime: Isolation: Resources:{CPUShares:0 Memory:0 NanoCPUs:0 CgroupParent: BlkioWeight:0 BlkioWeightDevice:[] BlkioDeviceReadBps:[] BlkioDeviceWriteBps:[] BlkioDeviceReadIOps:[] BlkioDeviceWriteIOps:[] CPUPeriod:0 CPUQuota:0 CPURealtimePeriod:0 CPURealtimeRuntime:0 CpusetCpus: CpusetMems: Devices:[] DeviceCgroupRules:[] DeviceRequests:[] KernelMemory:0 KernelMemoryTCP:0 MemoryReservation:0 MemorySwap:0 MemorySwappiness:<nil> OomKillDisable:<nil> PidsLimit:<nil> Ulimits:[] CPUCount:0 CPUPercent:0 IOMaximumIOps:0 IOMaximumBandwidth:0} Mounts:[{Type:volume Source:WORKFLOW-0e329888b1e68284519f1ef64a178c87d53d4736a250a86f58a7b733f7249ba4_JOB-default-hostname-env Target:/var/run/act ReadOnly:false Consistency: BindOptions:<nil> VolumeOptions:<nil> ImageOptions:<nil> TmpfsOptions:<nil> ClusterOptions:<nil>} {Type:volume Source:WORKFLOW-0e329888b1e68284519f1ef64a178c87d53d4736a250a86f58a7b733f7249ba4_JOB-default-hostname Target:/home/debian/.cache/act/893237c099a8aece/hostexecutor/act/runner/testdata ReadOnly:false Consistency: BindOptions:<nil> VolumeOptions:<nil> ImageOptions:<nil> TmpfsOptions:<nil> ClusterOptions:<nil>}] MaskedPaths:[] ReadonlyPaths:[] Init:<nil>}
2025-08-11T08:02:31.8870488Z [container-hostname/default-hostname] [WORKFLOW-0e329888b1e68284519f1ef64a178c87d53d4736a250a86f58a7b733f7249ba4_JOB-default-hostname-env] is not a valid volume, will be ignored

That explains why /var/run/act is not mounted:

Mounts:[{Type:volume Source:WORKFLOW-0e329888b1e68284519f1ef64a178c87d53d4736a250a86f58a7b733f7249ba4_JOB-default-hostname-env Target:/var/run/act ...

``` 2025-08-11T08:02:31.8870350Z [container-hostname/default-hostname] [DEBUG] Common container.HostConfig ==> &{Binds:[/var/run/docker.sock:/var/run/docker.sock] ContainerIDFile: LogConfig:{Type: Config:map[]} NetworkMode:WORKFLOW-0e329888b1e68284519f1ef64a178c87d53d4736a250a86f58a7b733f7249ba4_JOB-default-hostname-default-hostname-network PortBindings:map[] RestartPolicy:{Name: MaximumRetryCount:0} AutoRemove:true VolumeDriver: VolumesFrom:[] ConsoleSize:[0 0] Annotations:map[] CapAdd:[] CapDrop:[] CgroupnsMode: DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[] GroupAdd:[] IpcMode: Cgroup: Links:[] OomScoreAdj:0 PidMode: Privileged:false PublishAllPorts:false ReadonlyRootfs:false SecurityOpt:[] StorageOpt:map[] Tmpfs:map[] UTSMode: UsernsMode: ShmSize:0 Sysctls:map[] Runtime: Isolation: Resources:{CPUShares:0 Memory:0 NanoCPUs:0 CgroupParent: BlkioWeight:0 BlkioWeightDevice:[] BlkioDeviceReadBps:[] BlkioDeviceWriteBps:[] BlkioDeviceReadIOps:[] BlkioDeviceWriteIOps:[] CPUPeriod:0 CPUQuota:0 CPURealtimePeriod:0 CPURealtimeRuntime:0 CpusetCpus: CpusetMems: Devices:[] DeviceCgroupRules:[] DeviceRequests:[] KernelMemory:0 KernelMemoryTCP:0 MemoryReservation:0 MemorySwap:0 MemorySwappiness:<nil> OomKillDisable:<nil> PidsLimit:<nil> Ulimits:[] CPUCount:0 CPUPercent:0 IOMaximumIOps:0 IOMaximumBandwidth:0} Mounts:[{Type:volume Source:WORKFLOW-0e329888b1e68284519f1ef64a178c87d53d4736a250a86f58a7b733f7249ba4_JOB-default-hostname-env Target:/var/run/act ReadOnly:false Consistency: BindOptions:<nil> VolumeOptions:<nil> ImageOptions:<nil> TmpfsOptions:<nil> ClusterOptions:<nil>} {Type:volume Source:WORKFLOW-0e329888b1e68284519f1ef64a178c87d53d4736a250a86f58a7b733f7249ba4_JOB-default-hostname Target:/home/debian/.cache/act/893237c099a8aece/hostexecutor/act/runner/testdata ReadOnly:false Consistency: BindOptions:<nil> VolumeOptions:<nil> ImageOptions:<nil> TmpfsOptions:<nil> ClusterOptions:<nil>}] MaskedPaths:[] ReadonlyPaths:[] Init:<nil>} 2025-08-11T08:02:31.8870488Z [container-hostname/default-hostname] [WORKFLOW-0e329888b1e68284519f1ef64a178c87d53d4736a250a86f58a7b733f7249ba4_JOB-default-hostname-env] is not a valid volume, will be ignored ``` That explains why `/var/run/act` is not mounted: > `Mounts:[{Type:volume Source:WORKFLOW-0e329888b1e68284519f1ef64a178c87d53d4736a250a86f58a7b733f7249ba4_JOB-default-hostname-env Target:/var/run/act ...`
Author
Contributor

// add some default binds and mounts to ValidVolumes
rc.Config.ValidVolumes = append(rc.Config.ValidVolumes, name)
rc.Config.ValidVolumes = append(rc.Config.ValidVolumes, name+"-env")
// TODO: add a new configuration to control whether the docker daemon can be mounted

Seeing that all cases where this fail happens is with multiple containers, I suppose there is a race and the run context is modified by two containers, only one of which gets in the list.

https://code.forgejo.org/forgejo/runner/src/commit/555b322ce5922942e846c04a62925e3c09db89b7/act/runner/run_context.go#L174-L177 Seeing that all cases where this fail happens is with multiple containers, I suppose there is a race and the run context is modified by two containers, only one of which gets in the list.
Author
Contributor

Let's hope #849 fixed it and re-open if it surfaces again.

Let's hope https://code.forgejo.org/forgejo/runner/pulls/849 fixed it and re-open if it surfaces again.

A similar error just occurred in a private pipeline of mine, after updating docker on my system. But I am not sure if this is related or if I should open a separate issue.
Here is the trace output of it:

forgejo(version:v12.10.1)` received task 15 of job deploy, be triggered by event: push
workflow prepared
evaluating expression 'success()'
expression 'success()' evaluated to 'true'
🚀  Start image=node:22-bookworm
RunContext deploy is assigned random name 6a57ec9a31ae3d9143a4f702644ab9ac
platform not specified, defaulting to detected linux/amd64
Image exists? true
Cleaning up network for job deploy, and network name is: WORKFLOW-6a57ec9a31ae3d9143a4f702644ab9ac
[...]
  🐳  docker create image=node:22-bookworm platform=linux/amd64 entrypoint=["tail" "-f" "/dev/null"] cmd=[] network="WORKFLOW-6a57ec9a31ae3d9143a4f702644ab9ac"
Common container.Config ==> &{Hostname: Domainname: User: AttachStdin:false AttachStdout:false AttachStderr:false ExposedPorts:map[] Tty:false OpenStdin:false StdinOnce:false Env:[RUNNER_TOOL_CACHE=/opt/hostedtoolcache RUNNER_OS=Linux RUNNER_ARCH=X64 RUNNER_TEMP=/tmp LANG=C.UTF-8] Cmd:[] Healthcheck:<nil> ArgsEscaped:false Image:node:22-bookworm Volumes:map[] WorkingDir:/workspace/Wolkensteine/pages Entrypoint:[] NetworkDisabled:false MacAddress: OnBuild:[] Labels:map[] StopSignal: StopTimeout:<nil> Shell:[]}
Common container.HostConfig ==> &{Binds:[] ContainerIDFile: LogConfig:{Type: Config:map[]} NetworkMode:WORKFLOW-6a57ec9a31ae3d9143a4f702644ab9ac PortBindings:map[] RestartPolicy:{Name: MaximumRetryCount:0} AutoRemove:false VolumeDriver: VolumesFrom:[] ConsoleSize:[0 0] Annotations:map[] CapAdd:[] CapDrop:[] CgroupnsMode: DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[] GroupAdd:[] IpcMode: Cgroup: Links:[] OomScoreAdj:0 PidMode: Privileged:false PublishAllPorts:false ReadonlyRootfs:false SecurityOpt:[] StorageOpt:map[] Tmpfs:map[] UTSMode: UsernsMode: ShmSize:0 Sysctls:map[] Runtime: Isolation: Resources:{CPUShares:0 Memory:0 NanoCPUs:0 CgroupParent: BlkioWeight:0 BlkioWeightDevice:[] BlkioDeviceReadBps:[] BlkioDeviceWriteBps:[] BlkioDeviceReadIOps:[] BlkioDeviceWriteIOps:[] CPUPeriod:0 CPUQuota:0 CPURealtimePeriod:0 CPURealtimeRuntime:0 CpusetCpus: CpusetMems: Devices:[] DeviceCgroupRules:[] DeviceRequests:[] KernelMemory:0 KernelMemoryTCP:0 MemoryReservation:0 MemorySwap:0 MemorySwappiness:<nil> OomKillDisable:<nil> PidsLimit:<nil> Ulimits:[] CPUCount:0 CPUPercent:0 IOMaximumIOps:0 IOMaximumBandwidth:0} Mounts:[{Type:volume Source:6a57ec9a31ae3d9143a4f702644ab9ac-env Target:/var/run/act ReadOnly:false Consistency: BindOptions:<nil> VolumeOptions:<nil> ImageOptions:<nil> TmpfsOptions:<nil> ClusterOptions:<nil>} {Type:volume Source:6a57ec9a31ae3d9143a4f702644ab9ac Target:/workspace/Wolkensteine/pages ReadOnly:false Consistency: BindOptions:<nil> VolumeOptions:<nil> ImageOptions:<nil> TmpfsOptions:<nil> ClusterOptions:<nil>}] MaskedPaths:[] ReadonlyPaths:[] Init:0xc00032ed60}
input.NetworkAliases ==> [deploy]
Created container name=FORGEJO-ACTIONS-TASK-15_WORKFLOW-7eeba1d255ba89a052352dc4af270d2b161166eaccf604d02af265997a53365e_JOB-deploy id=36db14e16d8f59f73a0afa88cd43535a3cfa955c79db8a2d63917cce1b583062 from image node:22-bookworm (platform: linux/amd64)
ENV ==> [RUNNER_TOOL_CACHE=/opt/hostedtoolcache RUNNER_OS=Linux RUNNER_ARCH=X64 RUNNER_TEMP=/tmp LANG=C.UTF-8]
  🐳  docker run image=node:22-bookworm platform=linux/amd64 entrypoint=["tail" "-f" "/dev/null"] cmd=[] network="WORKFLOW-6a57ec9a31ae3d9143a4f702644ab9ac"
Starting container: 36db14e16d8f59f73a0afa88cd43535a3cfa955c79db8a2d63917cce1b583062
Started container: 36db14e16d8f59f73a0afa88cd43535a3cfa955c79db8a2d63917cce1b583062
Writing entry to tarball workflow/event.json len:5136
Writing entry to tarball workflow/envs.txt len:0
Extracting content to '/var/run/act/'
copyContent: failed to copy content to container: Error response from daemon: mkdirat var/run/act: path escapes from parent

The difference I noticed is the reason the copy operation failed, according to the docker daemon:
Could not find the file /var/run/act/ in container vs mkdirat var/run/act: path escapes from parent

A similar error just occurred in a private pipeline of mine, after updating docker on my system. But I am not sure if this is related or if I should open a separate issue. Here is the trace output of it: ``` forgejo(version:v12.10.1)` received task 15 of job deploy, be triggered by event: push workflow prepared evaluating expression 'success()' expression 'success()' evaluated to 'true' 🚀 Start image=node:22-bookworm RunContext deploy is assigned random name 6a57ec9a31ae3d9143a4f702644ab9ac platform not specified, defaulting to detected linux/amd64 Image exists? true Cleaning up network for job deploy, and network name is: WORKFLOW-6a57ec9a31ae3d9143a4f702644ab9ac [...] 🐳 docker create image=node:22-bookworm platform=linux/amd64 entrypoint=["tail" "-f" "/dev/null"] cmd=[] network="WORKFLOW-6a57ec9a31ae3d9143a4f702644ab9ac" Common container.Config ==> &{Hostname: Domainname: User: AttachStdin:false AttachStdout:false AttachStderr:false ExposedPorts:map[] Tty:false OpenStdin:false StdinOnce:false Env:[RUNNER_TOOL_CACHE=/opt/hostedtoolcache RUNNER_OS=Linux RUNNER_ARCH=X64 RUNNER_TEMP=/tmp LANG=C.UTF-8] Cmd:[] Healthcheck:<nil> ArgsEscaped:false Image:node:22-bookworm Volumes:map[] WorkingDir:/workspace/Wolkensteine/pages Entrypoint:[] NetworkDisabled:false MacAddress: OnBuild:[] Labels:map[] StopSignal: StopTimeout:<nil> Shell:[]} Common container.HostConfig ==> &{Binds:[] ContainerIDFile: LogConfig:{Type: Config:map[]} NetworkMode:WORKFLOW-6a57ec9a31ae3d9143a4f702644ab9ac PortBindings:map[] RestartPolicy:{Name: MaximumRetryCount:0} AutoRemove:false VolumeDriver: VolumesFrom:[] ConsoleSize:[0 0] Annotations:map[] CapAdd:[] CapDrop:[] CgroupnsMode: DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[] GroupAdd:[] IpcMode: Cgroup: Links:[] OomScoreAdj:0 PidMode: Privileged:false PublishAllPorts:false ReadonlyRootfs:false SecurityOpt:[] StorageOpt:map[] Tmpfs:map[] UTSMode: UsernsMode: ShmSize:0 Sysctls:map[] Runtime: Isolation: Resources:{CPUShares:0 Memory:0 NanoCPUs:0 CgroupParent: BlkioWeight:0 BlkioWeightDevice:[] BlkioDeviceReadBps:[] BlkioDeviceWriteBps:[] BlkioDeviceReadIOps:[] BlkioDeviceWriteIOps:[] CPUPeriod:0 CPUQuota:0 CPURealtimePeriod:0 CPURealtimeRuntime:0 CpusetCpus: CpusetMems: Devices:[] DeviceCgroupRules:[] DeviceRequests:[] KernelMemory:0 KernelMemoryTCP:0 MemoryReservation:0 MemorySwap:0 MemorySwappiness:<nil> OomKillDisable:<nil> PidsLimit:<nil> Ulimits:[] CPUCount:0 CPUPercent:0 IOMaximumIOps:0 IOMaximumBandwidth:0} Mounts:[{Type:volume Source:6a57ec9a31ae3d9143a4f702644ab9ac-env Target:/var/run/act ReadOnly:false Consistency: BindOptions:<nil> VolumeOptions:<nil> ImageOptions:<nil> TmpfsOptions:<nil> ClusterOptions:<nil>} {Type:volume Source:6a57ec9a31ae3d9143a4f702644ab9ac Target:/workspace/Wolkensteine/pages ReadOnly:false Consistency: BindOptions:<nil> VolumeOptions:<nil> ImageOptions:<nil> TmpfsOptions:<nil> ClusterOptions:<nil>}] MaskedPaths:[] ReadonlyPaths:[] Init:0xc00032ed60} input.NetworkAliases ==> [deploy] Created container name=FORGEJO-ACTIONS-TASK-15_WORKFLOW-7eeba1d255ba89a052352dc4af270d2b161166eaccf604d02af265997a53365e_JOB-deploy id=36db14e16d8f59f73a0afa88cd43535a3cfa955c79db8a2d63917cce1b583062 from image node:22-bookworm (platform: linux/amd64) ENV ==> [RUNNER_TOOL_CACHE=/opt/hostedtoolcache RUNNER_OS=Linux RUNNER_ARCH=X64 RUNNER_TEMP=/tmp LANG=C.UTF-8] 🐳 docker run image=node:22-bookworm platform=linux/amd64 entrypoint=["tail" "-f" "/dev/null"] cmd=[] network="WORKFLOW-6a57ec9a31ae3d9143a4f702644ab9ac" Starting container: 36db14e16d8f59f73a0afa88cd43535a3cfa955c79db8a2d63917cce1b583062 Started container: 36db14e16d8f59f73a0afa88cd43535a3cfa955c79db8a2d63917cce1b583062 Writing entry to tarball workflow/event.json len:5136 Writing entry to tarball workflow/envs.txt len:0 Extracting content to '/var/run/act/' copyContent: failed to copy content to container: Error response from daemon: mkdirat var/run/act: path escapes from parent ``` The difference I noticed is the reason the copy operation failed, according to the docker daemon: Could not find the file /var/run/act/ in container vs mkdirat var/run/act: path escapes from parent
Member
@Wolkensteine https://code.forgejo.org/forgejo/runner/issues/1524
Sign in to join this conversation.
No milestone
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
forgejo/runner#848
No description provided.