hashFiles fails with Unable to filepath.Walk: lstat /workspace/... #54

Open
opened 2023-07-14 16:19:00 +00:00 by earl-warren · 24 comments
Owner

Workaround

Description

on: [push]

jobs:
  test:
    runs-on: docker
    steps:

      - uses: actions/checkout@v3      
      - name: hashFiles
        run: |
          set -x
          echo "${{ hashFiles('README.md') }}"  

silently succeeds and displays nothing. But in the log it always fails with Unable to filepath.Walk: lstat /workspace/....

Example

https://codeberg.org/Kbin/kbin-core/actions/runs/9

image

### Workaround * Compute the hash in a previous step https://code.forgejo.org/forgejo/runner/issues/54#issuecomment-507 * Use the https://code.forgejo.org/actions/go-hashfiles Action that does the same ### Description ``` on: [push] jobs: test: runs-on: docker steps: - uses: actions/checkout@v3 - name: hashFiles run: | set -x echo "${{ hashFiles('README.md') }}" ``` silently succeeds and displays nothing. But in the log it always fails with `Unable to filepath.Walk: lstat /workspace/...`. ### Example https://codeberg.org/Kbin/kbin-core/actions/runs/9 ![image](/attachments/9a437bfa-d922-4c89-920c-a6acc01003e3)
Author
Owner

actions/cache#1 is about a similar error.

@rome-user do you confirm that it works for you with forgejo-runner? Can you provide a link to a working example? There may be a subtle difference that triggers this issue.

https://code.forgejo.org/actions/cache/issues/1 is about a similar error. @rome-user do you confirm that it works for you with `forgejo-runner`? Can you provide a link to a working example? There may be a subtle difference that triggers this issue.
Author
Owner

I was able to run the workflow locally with.

$ git clone https://codeberg.org/Kbin/kbin-core/
$ cd kbin-core
$  git fetch origin +refs/pull/*:refs/pull/*
$ git checkout ccdfd9028cbc909167cd91d0aac1c2af530fe334
$ forgejo-runner exec --workflows .forgejo/workflows/action.yaml
$ forgejo-runner --version
forgejo-runner version v2.3.0

I attached the logs of exec.

I was able to run the workflow locally with. ``` $ git clone https://codeberg.org/Kbin/kbin-core/ $ cd kbin-core $ git fetch origin +refs/pull/*:refs/pull/* $ git checkout ccdfd9028cbc909167cd91d0aac1c2af530fe334 $ forgejo-runner exec --workflows .forgejo/workflows/action.yaml $ forgejo-runner --version forgejo-runner version v2.3.0 ``` I attached the logs of `exec`.
56 KiB
Author
Owner

The logs of the runner you're running locally would help figure out why it does not work in your environment.

The logs of the runner you're running locally would help figure out why it does not work in your environment.

Running action on PR: https://codeberg.org/Kbin/kbin-core/pulls/755.

I'm not running my runner locally. I'm using Ubuntu Server 22.04, which runs the forgejo-runner, with Docker installed. After I'm also running GitLab on the same server.

@earl-warren See attachment for full log. Below a small snippet:

Jul 14 21:39:29 ubuntu-server forgejo-runner[1852327]: [Kbin CI/CD pipeline/build] [DEBUG] Working directory '/workspace/Kbin/kbin-core'
Jul 14 21:39:29 ubuntu-server forgejo-runner[1852327]: [Kbin CI/CD pipeline/build]   ❗  ::error::Input required and not supplied: key
Jul 14 21:39:29 ubuntu-server forgejo-runner[1852327]: [Kbin CI/CD pipeline/build]   | ::error::Input required and not supplied: key
Jul 14 21:39:29 ubuntu-server forgejo-runner[1852327]: [Kbin CI/CD pipeline/build]   ❌  Failure - Main https://code.forgejo.org/actions/cache@v3
Jul 14 21:39:29 ubuntu-server forgejo-runner[1852327]: [Kbin CI/CD pipeline/build] exitcode '1': failure
Running action on PR: https://codeberg.org/Kbin/kbin-core/pulls/755. I'm not running my runner locally. I'm using Ubuntu Server 22.04, which runs the `forgejo-runner`, with Docker installed. After I'm also running GitLab on the same server. @earl-warren See attachment for full log. Below a small snippet: ``` Jul 14 21:39:29 ubuntu-server forgejo-runner[1852327]: [Kbin CI/CD pipeline/build] [DEBUG] Working directory '/workspace/Kbin/kbin-core' Jul 14 21:39:29 ubuntu-server forgejo-runner[1852327]: [Kbin CI/CD pipeline/build] ❗ ::error::Input required and not supplied: key Jul 14 21:39:29 ubuntu-server forgejo-runner[1852327]: [Kbin CI/CD pipeline/build] | ::error::Input required and not supplied: key Jul 14 21:39:29 ubuntu-server forgejo-runner[1852327]: [Kbin CI/CD pipeline/build] ❌ Failure - Main https://code.forgejo.org/actions/cache@v3 Jul 14 21:39:29 ubuntu-server forgejo-runner[1852327]: [Kbin CI/CD pipeline/build] exitcode '1': failure ```
221 KiB
Author
Owner
Jul 14 21:39:29 ubuntu-server forgejo-runner[1852327]: [Kbin CI/CD pipeline/build] Unable to interpolate expression 'format('{0}-yarn-{1}', runner.os, hashFiles('**/yarn.lock'))': Unable to filepath.Walk: lstat /workspace/Kbin/kbin-core: no such file or directory

is causing the expression in

          key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}

to evaluate to an empty string, hence the error message.

``` Jul 14 21:39:29 ubuntu-server forgejo-runner[1852327]: [Kbin CI/CD pipeline/build] Unable to interpolate expression 'format('{0}-yarn-{1}', runner.os, hashFiles('**/yarn.lock'))': Unable to filepath.Walk: lstat /workspace/Kbin/kbin-core: no such file or directory ``` is causing the expression in ``` key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }} ``` to evaluate to an empty string, hence the error message.
earl-warren changed title from cache action: :error::Input required and not supplied: key to expression that fail to evaluate are only logged server side 2023-07-14 22:10:03 +00:00

This is a problem in hashFiles() I presume?

Ps. I do use the official example from GitHub cache action readme file.

This is a problem in `hashFiles()` I presume? Ps. I do use the official example from GitHub cache action readme file.

Ow I see.. by default your checkout location (part of checkout action) is not /workspace I think.. But something like /home/codeberg-runner/.cache, right.

But the cache action expects that working directory is /workspace for some reason?

Ow I see.. by default your checkout location (part of checkout action) is not `/workspace` I think.. But something like `/home/codeberg-runner/.cache`, right. But the cache action expects that working directory is `/workspace` for some reason?
Author
Owner

I think you're on the right track but that's unexpected: I would assume the evaluation happens with a current working directory that is the workspace. Is **/yarn.lock supposed to match ./yarn.lock as well? I can't remember right now.

I think you're on the right track but that's unexpected: I would assume the evaluation happens with a current working directory that is the workspace. Is `**/yarn.lock` supposed to match `./yarn.lock` as well? I can't remember right now.
Author
Owner

Can you share the config.yaml file that you have? If you have any.

Can you share the `config.yaml` file that you have? If you have any.
Author
Owner
on: [push]

jobs:
  test:
    runs-on: docker
    steps:

      - uses: actions/checkout@v3      
      - name: hashFiles
        run: |
          set -x
          echo "${{ hashFiles('README.md') }}"  

silently succeeds and displays nothing. But in the log it always fails with Unable to filepath.Walk: lstat /workspace/.... You are correct, it does not look where it should. The evaluation does not happen within the container and the workingdir is wrong. The problem goes back to 2.0.4 and maybe before, it is not new.

It works when run with docker exec because the configuration is set differently.

``` on: [push] jobs: test: runs-on: docker steps: - uses: actions/checkout@v3 - name: hashFiles run: | set -x echo "${{ hashFiles('README.md') }}" ``` silently succeeds and displays nothing. But in the log it always fails with `Unable to filepath.Walk: lstat /workspace/...`. You are correct, it does not look where it should. The evaluation does not happen within the container and the workingdir is wrong. The problem goes back to 2.0.4 and maybe before, it is not new. It works when run with `docker exec` because the configuration is set differently.
earl-warren changed title from expression that fail to evaluate are only logged server side to hashFiles fails with Unable to filepath.Walk: lstat /workspace/... 2023-07-14 23:40:56 +00:00

I think you're on the right track but that's unexpected: I would assume the evaluation happens with a current working directory that is the workspace. Is **/yarn.lock supposed to match ./yarn.lock as well? I can't remember right now.

Yes, I think so. The error is also more about the folder that doesn't exists (/workspace/Kbin/kbin-core according to the error).

Can you share the config.yaml file that you have? If you have any.

Sure, it's basically the default config. I can NOT upload files with yaml or yml extension to your issue ticket system... So I changed the extension to txt.. blehh.

Also I see that the ~/.cache/act folder is used for the cache and checkout actions:

image

Then there is also a ~/.cache/actcache, only containing a bolt.db file:

image

> I think you're on the right track but that's unexpected: I would assume the evaluation happens with a current working directory that is the workspace. Is `**/yarn.lock` supposed to match `./yarn.lock` as well? I can't remember right now. Yes, I think so. The error is also more about the folder that doesn't exists (`/workspace/Kbin/kbin-core` according to the error). > Can you share the `config.yaml` file that you have? If you have any. Sure, it's basically the default config. I can NOT upload files with `yaml` or `yml` extension to your issue ticket system... So I changed the extension to `txt`.. blehh. Also I see that the `~/.cache/act` folder is used for the cache and checkout actions: ![image](/attachments/5630fd29-5b70-4cc7-bbfd-c984048b2e35) Then there is also a `~/.cache/actcache`, only containing a bolt.db file: ![image](/attachments/f14ed26c-ceb8-4d30-9211-f268cdcf5b63)

silently succeeds and displays nothing. But in the log it always fails with Unable to filepath.Walk: lstat /workspace/.... You are correct, it does not look where it should. The evaluation does not happen within the container and the workingdir is wrong. The problem goes back to 2.0.4 and maybe before, it is not new.

You are using GitHub Actions (fork) actions. Which by default always use /workspace as working directory. Or on disk: /home/runner/work/, I think.

> silently succeeds and displays nothing. But in the log it always fails with `Unable to filepath.Walk: lstat /workspace/...`. You are correct, it does not look where it should. The evaluation does not happen within the container and the workingdir is wrong. The problem goes back to 2.0.4 and maybe before, it is not new. You are using GitHub Actions (fork) actions. Which by default always use `/workspace` as working directory. Or on disk: `/home/runner/work/`, I think.
Author
Owner

The hashFiles expression function is evaluated on the host where the runner resides. But it is given a WorkingDir that only exists within the container running the job. It is the only use of WorkingDir when evaluating expressions.

  • runner/internal/app/run/runner.go
           runnerConfig := &runner.Config{
           // On Linux, Workdir will be like "/<parent_directory>/<owner>/<repo>"
           // On Windows, Workdir will be like "\<parent_directory>\<owner>\<repo>"
           Workdir:        filepath.FromSlash(filepath.Clean(fmt.Sprintf("/%s/%s", r.cfg.Container.WorkdirParent, preset.Repository))),
           BindWorkdir:    false,
           ActionCacheDir: filepath.FromSlash(r.cfg.Host.WorkdirParent),
    
  • act/pkg/runner/run_context.go
      type RunContext struct {
        Name                string
        Config              *Config
    
  • act/pkg/runner/expression.go
          return expressionEvaluator{
                  interpreter: exprparser.NewInterpeter(ee, exprparser.Config{
                          Run:        rc.Run,
                          WorkingDir: rc.Config.Workdir,
                          Context:    "step",
                  }),
          }
    
  • act/pkg/exprparser/functions.go
      func (impl *interperterImpl) hashFiles(paths ...reflect.Value) (string, error) {
      ...
          if err := filepath.Walk(impl.config.WorkingDir, func(path string, fi fs.FileInfo, err..
    
The `hashFiles` expression function is evaluated on the host where the runner resides. But it is given a `WorkingDir` that only exists within the container running the job. It is the only use of `WorkingDir` when evaluating expressions. * runner/internal/app/run/runner.go ```go runnerConfig := &runner.Config{ // On Linux, Workdir will be like "/<parent_directory>/<owner>/<repo>" // On Windows, Workdir will be like "\<parent_directory>\<owner>\<repo>" Workdir: filepath.FromSlash(filepath.Clean(fmt.Sprintf("/%s/%s", r.cfg.Container.WorkdirParent, preset.Repository))), BindWorkdir: false, ActionCacheDir: filepath.FromSlash(r.cfg.Host.WorkdirParent), ``` * act/pkg/runner/run_context.go ```go type RunContext struct { Name string Config *Config ``` * act/pkg/runner/expression.go ``` return expressionEvaluator{ interpreter: exprparser.NewInterpeter(ee, exprparser.Config{ Run: rc.Run, WorkingDir: rc.Config.Workdir, Context: "step", }), } ``` * act/pkg/exprparser/functions.go ``` func (impl *interperterImpl) hashFiles(paths ...reflect.Value) (string, error) { ... if err := filepath.Walk(impl.config.WorkingDir, func(path string, fi fs.FileInfo, err.. ```
Author
Owner

This bug went undetected because it fails in a way that shows as a success instead of a failure. The test that verifies it works claims it does when in reality it aborts and reports success.

This bug went undetected because it fails in a way that shows as a success instead of a failure. The [test that verifies](https://code.forgejo.org/actions/setup-forgejo/src/branch/main/testdata/example-expression/.forgejo/workflows/test.yml#L101-L106) it works claims it does when in reality it aborts and reports success.

Ow.. I see. But then again, Docker shouldn't be aware of the host location on disk. RIght? So the hashFiles expression should become Docker aware somehow and search the file within the check-out location inside Docker.

If this works, I'm afraid what we might see after this step. Where will the cache be stored for Docker? After the run is completed, the docker container will be stopped/removed. So do we mount the cache folder inside docker, or?

Ow.. I see. But then again, Docker shouldn't be aware of the host location on disk. RIght? So the `hashFiles` expression should become Docker aware somehow and search the file within the check-out location inside Docker. If this works, I'm afraid what we might see after this step. Where will the cache be stored for Docker? After the run is completed, the docker container will be stopped/removed. So do we mount the cache folder inside docker, or?
Author
Owner

I think it is reasonable to assume the hashFiles is evaluated relative to the workspace directory, which is known to the runner because it is mounted within the container by default and can be shared with other contains created by docker actions. I'm not sure I see how it should be implemented properly though.

For now I'm afraid you'll need workaround to calculate the hash key. Such as introducing a step before to do that checksum and pass it to the next step.

I think it is reasonable to assume the `hashFiles` is evaluated relative to the workspace directory, which is known to the runner because it is mounted within the container by default and can be shared with other contains created by docker actions. I'm not sure I see how it should be implemented properly though. For now I'm afraid you'll need workaround to calculate the hash key. Such as introducing a step before to do that checksum and pass it to the next step.
Author
Owner

The cache is stored on the host, that works.

The cache is stored on the host, that works.

Yup that works:

image

For others that might need this:

      - name: Calculate yarn.lock hash
        id: yarn-lock-hash
        run: |
          echo "hash=$(md5sum yarn.lock)" >> $GITHUB_OUTPUT          

      - name: Calculate composer.lock hash
        id: composer-lock-hash
        run: |
          echo "hash=$(md5sum composer.lock)" >> $GITHUB_OUTPUT          

And then I use this:

      - uses: https://code.forgejo.org/actions/cache@v3
        id: yarn-cache # use this to check for `cache-hit` (`steps.yarn-cache.outputs.cache-hit != 'true'`)
        with:
          path: ${{ steps.yarn-cache-dir-path.outputs.dir }}
          key: ${{ runner.os }}-yarn-${{ steps.yarn-lock-hash.outputs.hash }}
          restore-keys: ${{ runner.os }}-yarn-

      - uses: https://code.forgejo.org/actions/cache@v3
        with:
          path: ${{ steps.composer-cache.outputs.dir }}
          key: ${{ runner.os }}-composer-${{ steps.composer-lock-hash.outputs.hash }}
          restore-keys: ${{ runner.os }}-composer-
Yup that works: ![image](/attachments/39dfcbc1-59a0-4ed4-934c-261e4b319d36) For others that might need this: ```yaml - name: Calculate yarn.lock hash id: yarn-lock-hash run: | echo "hash=$(md5sum yarn.lock)" >> $GITHUB_OUTPUT - name: Calculate composer.lock hash id: composer-lock-hash run: | echo "hash=$(md5sum composer.lock)" >> $GITHUB_OUTPUT ``` And then I use this: ``` - uses: https://code.forgejo.org/actions/cache@v3 id: yarn-cache # use this to check for `cache-hit` (`steps.yarn-cache.outputs.cache-hit != 'true'`) with: path: ${{ steps.yarn-cache-dir-path.outputs.dir }} key: ${{ runner.os }}-yarn-${{ steps.yarn-lock-hash.outputs.hash }} restore-keys: ${{ runner.os }}-yarn- - uses: https://code.forgejo.org/actions/cache@v3 with: path: ${{ steps.composer-cache.outputs.dir }} key: ${{ runner.os }}-composer-${{ steps.composer-lock-hash.outputs.hash }} restore-keys: ${{ runner.os }}-composer- ```

That being said. I hope this hashFiles method can fixed in the future hopefully.

That being said. I hope this `hashFiles` method can fixed in the future hopefully.
Member

I modified the description with @melroy89 workaround and @earl-warren description so others do not need to read the whole thread to figure that out.

I modified the description with @melroy89 workaround and @earl-warren description so others do not need to read the whole thread to figure that out.
dachary added the
Kind/Bug
Priority
High
labels 2023-07-15 15:14:41 +00:00

Stumbled upon this.
I'm trying to run with forgejo-runner exec and hashFiles returns empty:

jobs:
  myfile:
    runs-on: docker
    steps:
      - name: generate output
        run: |
          echo "this is a file" > myfile
      - name: ensure file is there
        run: |
          ls
          cat myfile
      - name: test output
        run: |
          echo "${{ hashFiles('myfile') || 'none' }}"

Also, not sure if its related (if its not, I can create a new issue if its a bug):
If a file already exists in the working directory when running forgejo-runner, hashFiles works as expected. If the file has been created in the working directory along the workflow (even different steps) then it doesn't work.

Stumbled upon this. I'm trying to run with `forgejo-runner exec` and hashFiles returns empty: ``` jobs: myfile: runs-on: docker steps: - name: generate output run: | echo "this is a file" > myfile - name: ensure file is there run: | ls cat myfile - name: test output run: | echo "${{ hashFiles('myfile') || 'none' }}" ``` Also, not sure if its related (if its not, I can create a new issue if its a bug): If a file already exists in the working directory when running forgejo-runner, hashFiles works as expected. If the file has been created in the working directory along the workflow (even different steps) then it doesn't work.
Author
Owner

I think that's because hashFiles is evaluated before the job starts running. There is non trivial work to do to implement hashFiles I'm afraid.

I think that's because hashFiles is evaluated **before** the job starts running. There is non trivial work to do to implement hashFiles I'm afraid.

@earl-warren When reading the debug output, It writes that evaluates the expression on the time that it executes the step, but I haven't looked at the source to tell for sure. If it is the case I completely understand.

For now I've replaced with the solution mentioned above to use md5sum/sha256sum among the steps.

Using the workflow above:

[test-simple.yml/myfile]   �  docker exec cmd=[bash --noprofile --norc -e -o pipefail /var/run/act/workflow/1] user= workdir=
[test-simple.yml/myfile] [DEBUG] Exec command '[bash --noprofile --norc -e -o pipefail /var/run/act/workflow/1]'
[test-simple.yml/myfile] [DEBUG] Working directory '/home/diem/carpediem/repos/containers/test-forgejo/mycontainer'
| myfile
| this is a file
[test-simple.yml/myfile]   ✅  Success - Main ensure file is there
....
[test-simple.yml/myfile] [DEBUG] expression 'echo "${{ hashFiles('myfile') || 'none' }}"' rewritten to 'format('echo "{0}"', hashFiles('myfile') || 'none')'
[test-simple.yml/myfile] [DEBUG] evaluating expression 'format('echo "{0}"', hashFiles('myfile') || 'none')'
[test-simple.yml/myfile] [DEBUG] expression 'format('echo "{0}"', hashFiles('myfile') || 'none')' evaluated to '%!t(string=echo "none")'
[test-simple.yml/myfile] [DEBUG] Wrote add-mask command to 'workflow/2'
[test-simple.yml/myfile] [DEBUG] Writing entry to tarball workflow/2 len:13
[test-simple.yml/myfile] [DEBUG] Extracting content to '/var/run/act'
[test-simple.yml/myfile]   �  docker exec cmd=[bash --noprofile --norc -e -o pipefail /var/run/act/workflow/2] user= workdir=
[test-simple.yml/myfile] [DEBUG] Exec command '[bash --noprofile --norc -e -o pipefail /var/run/act/workflow/2]'
[test-simple.yml/myfile] [DEBUG] Working directory '/home/diem/carpediem/repos/containers/test-forgejo/mycontainer'
| none
[test-simple.yml/myfile]   ✅  Success - Main test output

What I thought it could be related if hashFiles isn't being called from the exactly same directory as the container mounts as volume, thus not "seeing" any file created by the steps before it.

@earl-warren When reading the debug output, It writes that evaluates the expression on the time that it executes the step, but I haven't looked at the source to tell for sure. If it is the case I completely understand. For now I've replaced with the solution mentioned above to use md5sum/sha256sum among the steps. Using the workflow above: ``` [test-simple.yml/myfile] � docker exec cmd=[bash --noprofile --norc -e -o pipefail /var/run/act/workflow/1] user= workdir= [test-simple.yml/myfile] [DEBUG] Exec command '[bash --noprofile --norc -e -o pipefail /var/run/act/workflow/1]' [test-simple.yml/myfile] [DEBUG] Working directory '/home/diem/carpediem/repos/containers/test-forgejo/mycontainer' | myfile | this is a file [test-simple.yml/myfile] ✅ Success - Main ensure file is there .... [test-simple.yml/myfile] [DEBUG] expression 'echo "${{ hashFiles('myfile') || 'none' }}"' rewritten to 'format('echo "{0}"', hashFiles('myfile') || 'none')' [test-simple.yml/myfile] [DEBUG] evaluating expression 'format('echo "{0}"', hashFiles('myfile') || 'none')' [test-simple.yml/myfile] [DEBUG] expression 'format('echo "{0}"', hashFiles('myfile') || 'none')' evaluated to '%!t(string=echo "none")' [test-simple.yml/myfile] [DEBUG] Wrote add-mask command to 'workflow/2' [test-simple.yml/myfile] [DEBUG] Writing entry to tarball workflow/2 len:13 [test-simple.yml/myfile] [DEBUG] Extracting content to '/var/run/act' [test-simple.yml/myfile] � docker exec cmd=[bash --noprofile --norc -e -o pipefail /var/run/act/workflow/2] user= workdir= [test-simple.yml/myfile] [DEBUG] Exec command '[bash --noprofile --norc -e -o pipefail /var/run/act/workflow/2]' [test-simple.yml/myfile] [DEBUG] Working directory '/home/diem/carpediem/repos/containers/test-forgejo/mycontainer' | none [test-simple.yml/myfile] ✅ Success - Main test output ``` What I thought it could be related if `hashFiles` isn't being called from the exactly same directory as the container mounts as volume, thus not "seeing" any file created by the steps before it.

actions/cache#1 is about a similar error.

@rome-user do you confirm that it works for you with forgejo-runner? Can you provide a link to a working example? There may be a subtle difference that triggers this issue.

Hi, I tested actions/cache just now using forgejo/runner:2.3.0 with rootless docker-in-docker. I can confirm that the issue I closed was not fixed by using forgejo runner.

> https://code.forgejo.org/actions/cache/issues/1 is about a similar error. > > @rome-user do you confirm that it works for you with `forgejo-runner`? Can you provide a link to a working example? There may be a subtle difference that triggers this issue. Hi, I tested actions/cache just now using `forgejo/runner:2.3.0` with rootless docker-in-docker. I can confirm that the issue I closed was not fixed by using forgejo runner.
Sign in to join this conversation.
No milestone
No project
No assignees
5 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: forgejo/runner#54
No description provided.