forgejo/runner
Watch
49
112
Fork
You've already forked runner
 111
Code Issues 38 Pull requests 15 Releases 98 Packages 1 Activity Actions 2

bug: actions/cache has inconsistent behavior in LXC and Docker containers. #918

New issue
Closed
opened 2025-08-28 17:33:47 +00:00 by Hill-98 · 17 comments
Hill-98 commented 2025-08-28 17:33:47 +00:00

Can you reproduce the bug on the Forgejo test instance?

No

Description

This is an extension of issue #916

I ran the following workflow twice and the LXC container always runs with Cache not found for input keys: lxc-cache, but if I remove the ${{ forge.workspace }}/ from the path parameter, it works fine.

either way, after the workflow execution is complete, I see the message in the output that the cache was saved.

my runner config:

{
  "WARNING": "This file is automatically generated by act-runner. Do not edit it manually unless you know what you are doing. Removing this file will cause act runner to re-register as a new runner.",
  "id": 3,
  "uuid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
  "name": "forgejo-runner",
  "token": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
  "address": "https://git.xxx.fun/",
  "labels": [
    "docker:docker://catthehacker/ubuntu:act-latest",
    "ubuntu-latest:docker://catthehacker/ubuntu:act-latest",
    "lxc:lxc://debian:bookworm"
  ]
}

Forgejo Version

12.0.1+gitea-1.22.0

Runner Version

9.1.1

How are you running Forgejo?

Docker

How are you running the Runner?

Debian Bookworm on QEMU VM

Logs

No response

Workflow file

name: actions/cache test

on:
  workflow_dispatch:

jobs:
  docker-cache-test:
    runs-on: docker
    steps:
      - name: cache test
        id: cache
        uses: actions/cache@v4
        with:
          path: ${{ forge.workspace }}/cache
          key: docker-cache

      - name: write cache
        if: ${{ steps.cache.outputs.cache-hit != 'true' }}
        run: date > ${{ forge.workspace }}/cache

      - name: read cacha
        run: cat ${{ forge.workspace }}/cache
  lxc-cache-test:
    runs-on: lxc
    steps:
      - name: cache test
        id: cache
        uses: actions/cache@v4
        with:
          path: ${{ forge.workspace }}/cache
          key: lxc-cache

      - name: write cache
        if: ${{ steps.cache.outputs.cache-hit != 'true' }}
        run: date > ${{ forge.workspace }}/cache

      - name: read cacha
        run: cat ${{ forge.workspace }}/cache
### Can you reproduce the bug on the Forgejo test instance? No ### Description This is an extension of issue #916 I ran the following workflow twice and the LXC container always runs with `Cache not found for input keys: lxc-cache`, but if I remove the `${{ forge.workspace }}/` from the `path` parameter, it works fine. either way, after the workflow execution is complete, I see the message in the output that the cache was saved. my runner config: ```json { "WARNING": "This file is automatically generated by act-runner. Do not edit it manually unless you know what you are doing. Removing this file will cause act runner to re-register as a new runner.", "id": 3, "uuid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "name": "forgejo-runner", "token": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "address": "https://git.xxx.fun/", "labels": [ "docker:docker://catthehacker/ubuntu:act-latest", "ubuntu-latest:docker://catthehacker/ubuntu:act-latest", "lxc:lxc://debian:bookworm" ] } ``` ### Forgejo Version 12.0.1+gitea-1.22.0 ### Runner Version 9.1.1 ### How are you running Forgejo? Docker ### How are you running the Runner? Debian Bookworm on QEMU VM ### Logs _No response_ ### Workflow file ```yaml name: actions/cache test on: workflow_dispatch: jobs: docker-cache-test: runs-on: docker steps: - name: cache test id: cache uses: actions/cache@v4 with: path: ${{ forge.workspace }}/cache key: docker-cache - name: write cache if: ${{ steps.cache.outputs.cache-hit != 'true' }} run: date > ${{ forge.workspace }}/cache - name: read cacha run: cat ${{ forge.workspace }}/cache lxc-cache-test: runs-on: lxc steps: - name: cache test id: cache uses: actions/cache@v4 with: path: ${{ forge.workspace }}/cache key: lxc-cache - name: write cache if: ${{ steps.cache.outputs.cache-hit != 'true' }} run: date > ${{ forge.workspace }}/cache - name: read cacha run: cat ${{ forge.workspace }}/cache ```
earl-warren commented 2025-08-29 11:15:45 +00:00
Contributor

Could you please attach the logs?

Could you please attach the logs?
earl-warren commented 2025-08-29 11:17:17 +00:00
Contributor

This is a good reproducer, I'll try that today.

This is a good reproducer, I'll try that today.
earl-warren commented 2025-08-29 11:47:57 +00:00
Contributor

I copy/pasted the workflow verbatim. With a runner 9.1.1 installed from binary with the following config, it works for me 🤔

log:
  level: trace
  job_level: trace
runner:
  file: .runner
  capacity: 1
  envs:
    A_TEST_ENV_NAME_1: a_test_env_value_1
    A_TEST_ENV_NAME_2: a_test_env_value_2
  env_file: .env
  timeout: 3h
  shutdown_timeout: 3h
  insecure: false
  fetch_timeout: 5s
  fetch_interval: 2s
  report_interval: 1s
  labels: ["ubuntu-latest:docker://node:20-bookworm","docker:docker://node:20-bookworm", "lxc:lxc://debian:bookworm"]
cache:
  enabled: true
  dir: ""
  host: ""
  port: 0
  proxy_port: 0
  external_server: ""
  secret: ""
  actions_cache_url_override: ""
container:
  network: ""
  enable_ipv6: false
  privileged: false
  options:
  workdir_parent:
  valid_volumes: []
  docker_host: "-"
  force_pull: false
  force_rebuild: false
host:
  workdir_parent:

https://v13.next.forgejo.org/earl-warren/lxc-cache/actions/runs/1/jobs/1

image

I copy/pasted the workflow verbatim. With a runner 9.1.1 installed from binary with the following config, it works for me 🤔 ```yaml log: level: trace job_level: trace runner: file: .runner capacity: 1 envs: A_TEST_ENV_NAME_1: a_test_env_value_1 A_TEST_ENV_NAME_2: a_test_env_value_2 env_file: .env timeout: 3h shutdown_timeout: 3h insecure: false fetch_timeout: 5s fetch_interval: 2s report_interval: 1s labels: ["ubuntu-latest:docker://node:20-bookworm","docker:docker://node:20-bookworm", "lxc:lxc://debian:bookworm"] cache: enabled: true dir: "" host: "" port: 0 proxy_port: 0 external_server: "" secret: "" actions_cache_url_override: "" container: network: "" enable_ipv6: false privileged: false options: workdir_parent: valid_volumes: [] docker_host: "-" force_pull: false force_rebuild: false host: workdir_parent: ``` https://v13.next.forgejo.org/earl-warren/lxc-cache/actions/runs/1/jobs/1 ![image](/attachments/9b15b9e3-c326-4fab-9717-4cecf8b17884)
image.png
74 KiB
Hill-98 commented 2025-08-29 15:22:13 +00:00
Author

The second time the workflow is running, LXC still has no cache found.

Is the same result of your test?

The second time the workflow is running, LXC still has no cache found. Is the same result of your test?
Hill-98 commented 2025-08-29 15:27:40 +00:00
Author

My previous wording may not have been very clear. If the workflow is run twice, the Docker container can restore the cache on the second run, but the LXC container always outputs "cache not found."

My previous wording may not have been very clear. If the workflow is run twice, the Docker container can restore the cache on the second run, but the LXC container always outputs "cache not found."
earl-warren commented 2025-08-29 16:21:15 +00:00
Contributor

I get it now.

First run

https://v13.next.forgejo.org/earl-warren/lxc-cache/actions/runs/1/jobs/1

Both Docker & LXC populate the cache

image

Second run

Docker retrieves the cache successfully

image

LXC does not retrieve the cache successfully

image

I get it now. ## First run https://v13.next.forgejo.org/earl-warren/lxc-cache/actions/runs/1/jobs/1 Both Docker & LXC populate the cache ![image](/attachments/9b15b9e3-c326-4fab-9717-4cecf8b17884) ## Second run [Docker retrieves the cache successfully](https://v13.next.forgejo.org/earl-warren/lxc-cache/actions/runs/2/jobs/0) ![image](/attachments/05eb33c0-00ef-416a-a39f-1c7836aba1ca) [LXC does not retrieve the cache successfully](https://v13.next.forgejo.org/earl-warren/lxc-cache/actions/runs/2/jobs/1) ![image](/attachments/e0740a13-206d-4797-8616-381aeecb4948)
image.png
86 KiB
image.png
69 KiB
earl-warren commented 2025-08-29 16:28:23 +00:00
Contributor

With this workflow (i.e. path: cache instead of path: ${{ forge.workspace }}/cache)

name: actions/cache test

on:
  workflow_dispatch:

jobs:
  lxc-cache-test:
    runs-on: lxc
    steps:
      - name: cache test
        id: cache
        uses: actions/cache@v4
        with:
          path: cache
          key: lxc-cache

      - name: write cache
        if: ${{ steps.cache.outputs.cache-hit != 'true' }}
        run: date > ${{ forge.workspace }}/cache

      - name: read cacha
        run: cat ${{ forge.workspace }}/cache

First run

Populates the cache

https://v13.next.forgejo.org/earl-warren/lxc-cache/actions/runs/3

image

Second run

Restores the cache

https://v13.next.forgejo.org/earl-warren/lxc-cache/actions/runs/4

image

With this workflow (i.e. `path: cache` instead of `path: ${{ forge.workspace }}/cache`) ```yaml name: actions/cache test on: workflow_dispatch: jobs: lxc-cache-test: runs-on: lxc steps: - name: cache test id: cache uses: actions/cache@v4 with: path: cache key: lxc-cache - name: write cache if: ${{ steps.cache.outputs.cache-hit != 'true' }} run: date > ${{ forge.workspace }}/cache - name: read cacha run: cat ${{ forge.workspace }}/cache ``` ## First run Populates the cache https://v13.next.forgejo.org/earl-warren/lxc-cache/actions/runs/3 ![image](/attachments/ca580772-bc87-4a25-8472-a297a0f88deb) ## Second run Restores the cache https://v13.next.forgejo.org/earl-warren/lxc-cache/actions/runs/4 ![image](/attachments/21543b0e-1141-49ea-b575-30fae602fba4)
image.png
70 KiB
image.png
91 KiB
earl-warren commented 2025-08-29 16:41:58 +00:00
Contributor

@Hill-98 it is perfectly reproducible now 👍

@Hill-98 it is perfectly reproducible now 👍
👍 1
earl-warren commented 2025-09-16 13:53:56 +00:00
Contributor

There is no guarantee that ${{ forge.workspace }} is a constant. It turns out to be for Docker/podman based jobs (e.g. /workspace/earl-warren/lxc-cache). But for LXC based jobs it contains a random path component (e.g. /home/earl-warren/.cache/act/9dc369b9a28961d6/hostexecutor where 9dc369b9a28961d6 is the random part).

This is why the second run cannot retrieve the cache from the previous run: they both have different paths. And also the reason why it succeeds when using cache alone, because it is the same from one run to the next.

Does that clarify the issue?

There is no guarantee that `${{ forge.workspace }}` is a constant. It turns out to be for Docker/podman based jobs (e.g. `/workspace/earl-warren/lxc-cache`). But for LXC based jobs it contains a random path component (e.g. `/home/earl-warren/.cache/act/9dc369b9a28961d6/hostexecutor` where `9dc369b9a28961d6` is the random part). This is why the second run cannot retrieve the cache from the previous run: they both have different paths. And also the reason why it succeeds when using `cache` alone, because it is the same from one run to the next. Does that clarify the issue?
Hill-98 commented 2025-09-16 14:17:00 +00:00
Author

I see. So would changing actions/cache or the cache server fix this?

I see. So would changing `actions/cache` or the cache server fix this?
earl-warren commented 2025-09-16 15:03:11 +00:00
Contributor

I acknowledge this is confusing (I was confused myself... ) because it is not immediately intuitive that ${{ forge.workspace }}/mydirectory may be evaluated to a different string on each job run. It should be documented.

But it does not seem like a bug to fix. You will get a consistent behavior if path: something/I/want/to/cache is identical across job runs.

Do you see what I mean?

I acknowledge this is confusing (I was confused myself... ) because it is not immediately intuitive that `${{ forge.workspace }}/mydirectory` may be evaluated to a different string on each job run. It should be documented. But it does not seem like a bug to fix. You will get a consistent behavior if `path: something/I/want/to/cache` is identical across job runs. Do you see what I mean?
Hill-98 commented 2025-09-16 15:09:39 +00:00
Author

I see.

Do you think we should record this in the documentation?

I see. Do you think we should record this in the documentation?
earl-warren commented 2025-09-16 16:09:51 +00:00
Contributor

I thought about that but... actions/cache is where it would belong to be discoverable. Or even as a bug report there: the root of my confusion (and yours presumably) is that I don't understand why the path would matter to retrieve the entry when the key is unambiguous. But it does, evidently.

I thought about that but... `actions/cache` is where it would belong to be discoverable. Or even as a bug report there: the root of my confusion (and yours presumably) is that I don't understand why the `path` would matter to retrieve the entry when the `key` is unambiguous. But it does, evidently.
Hill-98 commented 2025-09-16 19:18:35 +00:00
Author

Look at: https://github.com/actions/cache#cache-version

The version of the cache used internally is related to the path, so this is expected for GitHub, but broken for Forgejo's LXC.

Look at: https://github.com/actions/cache#cache-version The version of the cache used internally is related to the path, so this is expected for GitHub, but broken for Forgejo's LXC.
earl-warren commented 2025-09-16 20:39:58 +00:00
Contributor

The version of the cache used internally is related to the path,...

Good find, I missed it!

, so this is expected for GitHub, but broken for Forgejo's LXC.

It works as expected if the path is not random. ${{ forge.workspace }} is not guaranteed to be a fixed string across all jobs. It happens to be for Docker based runs in Forgejo Actions. But it is not for LXC based runs.

What is the problem with using something instead of ${{ forge.workspace }}/something for the path?

> The version of the cache used internally is related to the path,... Good find, I missed it! > , so this is expected for GitHub, but broken for Forgejo's LXC. It works as expected if the path is not random. `${{ forge.workspace }}` is not guaranteed to be a fixed string across all jobs. It happens to be for Docker based runs in Forgejo Actions. But it is not for LXC based runs. What is the problem with using `something` instead of `${{ forge.workspace }}/something` for the path?
Hill-98 commented 2025-09-17 12:11:12 +00:00
Author

What is the problem with using something instead of ${{ forge.workspace }}/something for the path?

No problem, but the absolute path is always reassuring, lol...

In my current use case, this problem no longer affects me.

> What is the problem with using something instead of ${{ forge.workspace }}/something for the path? No problem, but the absolute path is always reassuring, lol... In my current use case, this problem no longer affects me.
👍 1
earl-warren commented 2025-09-17 12:46:35 +00:00
Contributor

I agree absolute paths are reassuring. /opt/cache would also do.

Let's close this then.

I agree absolute paths are reassuring. `/opt/cache` would also do. Let's close this then.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
renovate/github.com-docker-cli-29.x
renovate/buildx-0.x
enable-namespaces
support-v12.13.x
renovate/forgejo-lxc-13.x
validate-1460
chore/testing-trixie
chore/testing-bookworm
expand-reusable-workflows-working-branch
feat-default_actions_url-on-step
2023-05-22-main-before-force-push
2023-04-30-main-before-force-push
v13.0.0
v12.13.2
v12.13.1
v12.13.0
v12.12.0
v12.11.1
v12.11.0
v12.10.2
v12.10.1
v12.10.0
v12.9.0
v12.8.2
v12.8.1
v12.8.0
v12.7.3
v12.7.2
v12.7.1
v12.7.0
v12.6.4
v12.6.3
v12.6.2
v12.6.1
v12.6.0
v12.5.3
v12.5.2
v12.5.1
v12.5.0
v12.4.0
v12.3.1
v12.3.0
v12.2.0
v12.1.2
v12.1.1
v12.1.0
v12.0.1
v12.0.0
v11.3.1
v11.3.0
v11.2.0
v11.1.2
v11.1.1
v11.1.0
v11.0.0
v10.0.1
v10.0.0
v9.1.1
v9.1.0
v9.0.3
v9.0.2
v9.0.1
v9.0.0
v8.0.1
v8.0.0
v7.0.0
v6.4.0
v6.3.1
v6.3.0
v6.2.2
v6.2.1
v6.2.0
v6.1.0
v6.0.1
v6.0.0
v5.0.4
v5.0.3
v5.0.2
v5.0.1
v5.0.0
v4.0.1
v4.0.0
v3.5.1
v3.5.0
v3.4.1
v3.4.0
v3.3.0
v3.2.0
v3.1.0
v3.0.3
v3.0.2
v3.0.1
v3.0.0
v2.5.0
v2.4.0
v2.3.0
v2.2.0
v2.1.0
v2.0.4
v2.0.3
v2.0.2
v1.5.0
v2.0.1
v2.0.0
v1.8.1
v1.8.0
v1.7.0
v1.6.0
v1.4.3
v1.4.2
v1.4.1
v1.3.1
v1.4.0
v1.3.0
v1.2.0
v1.1.0
v0.0.1
v1.0.1
v1.0.0
Labels
Clear labels   
FreeBSD

   
Kind/Breaking
Breaking change that won't be backward compatible

  
Kind/Bug
Something is not working

  
Kind/Chore
It is no fun but it needs to be done, this is a chore.

  
Kind/DependencyUpdate
for pull requests that are created by renovate when updating a dependency

  
Kind/Documentation
Documentation changes

  
Kind/Enhancement
Improve existing functionality

  
Kind/Feature
New functionality

  
Kind/Security
This is security issue

  
Kind/Testing
Issue or pull request related to testing

  
Priority
Critical
 The priority is critical

  
Priority
High
 The priority is high

  
Priority
Low
 The priority is low

  
Priority
Medium
 The priority is medium

  
Reviewed
Confirmed
 Issue has been confirmed

  
Reviewed
Duplicate
 This issue or pull request already exists

  
Reviewed
Invalid
 Invalid issue

  
Reviewed
Won't Fix
 This issue won't be fixed

  
Status
Abandoned
 Somebody has started to work on this but abandoned work

  
Status
Blocked
 Something is blocking this issue or pull request

  
Status
Need More Info
 Feedback is required to reproduce issue or to continue work

  
Windows
Windows ports is an independent project, see https://github.com/Crown0815/Forgejo-runner-windows-builder

  
linux-powerpc64le
powerpc64le

  
linux-riscv64

   
linux-s390x

   
run-end-to-end-tests
Run https://code.forgejo.org/forgejo/end-to-end with a runner compiled from this pull request

  
run-forgejo-tests
Run https://codeberg.org/forgejo/forgejo tests importing the runner from this pull request

  
run-multi-platform-tests
Label a PR for automatic execution of test suite on arm64 architecture.

No labels
FreeBSD
Kind/Breaking
Kind/Bug
Kind/Chore
Kind/DependencyUpdate
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
Windows
linux-powerpc64le
linux-riscv64
linux-s390x
run-end-to-end-tests
run-forgejo-tests
run-multi-platform-tests
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
forgejo/runner#918
No description provided.