artifacts can no longer be uploaded on a repository #181

New issue

Open

opened 2024-04-17 21:48:57 +00:00 by earl-warren · 3 comments

earl-warren commented

2024-04-17 21:48:57 +00:00

Owner

Description

Re-running a workflow for which an artifact was already successfully uploaded failed with the upload action logging the following. There were no error messages server side. The same workflow in the same instance successfully uploaded artifacts an hour before.

New runs of the same workflow now permanently fail to upload artifacts with the same behavior.

Forgejo Version

7.0.0-dev-2136-f075579c95+gitea-1.22.0

Can you reproduce the bug on Forgejo Next?

Logs

[/build]   | ::debug::Root artifact directory is /tmp/forgejo-upload
[/build]   | Starting artifact upload
[/build]   | For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-s\
tep-debug-logging
Artifact name is valid!
[/build]   ??  ::debug::Artifact Url: https://code.forgejo.org/api/actions_pipeline/_apis/pipelines/workflows/16640/artifacts?api-version=6.0-preview
[/build]   | ::debug::Artifact Url: https://code.forgejo.org/api/actions_pipeline/_apis/pipelines/workflows/16640/artifacts?api-version=6.0-preview
[/build]   ??  ::debug::Upload Resource URL: https://code.forgejo.org/api/actions_pipeline/_apis/pipelines/workflows/16640/artifacts/6eff8731602e12a72be3802a16f1d0c9/upload
[/build]   | ::debug::Upload Resource URL: https://code.forgejo.org/api/actions_pipeline/_apis/pipelines/workflows/16640/artifacts/6eff8731602e12a72be3802a16f1d0c9/upload
  | ExponentiContainer for artifact "forgejo-dev" successfully created. Starting upload of file(s)
[/build]   ??  ::debug::File Concurrency: 2, and Chunk Size: 8388608
[/build]   | ::debug::File Concurrency: 2, and Chunk Size: 8388608
[/build]   ??  ::debug::/tmp/forgejo-upload/forgejo-7.0-dev is greater than 64k in size. Creating a gzip file on-disk /tmp/tmp-21867-AS36ytViavti to potentially reduce the upload size
[/build]   | ::debug::/tmp/forgejo-upload/forgejo-7.0-dev is greater than 64k in size. Creating a gzip file on-disk /tmp/tmp-21867-AS36ytViavti to potentially reduce the upload size
[/build]   ??  ::debug::/tmp/forgejo-upload/forgejo-8.0-dev is greater than 64k in size. Creating a gzip file on-disk /tmp/tmp-21867-YvEbmfwK2uho to potentially reduce the upload size
[/build]   | ::debug::/tmp/forgejo-upload/forgejo-8.0-dev is greater than 64k in size. Creating a gzip file on-disk /tmp/tmp-21867-YvEbmfwK2uho to potentially reduce the upload size
[/build]   ??  ::debug::The gzip file created for /tmp/forgejo-upload/forgejo-8.0-dev is smaller than the original file. The file will be uploaded using gzip.
[/build]   | ::debug::The gzip file created for /tmp/forgejo-upload/forgejo-8.0-dev is smaller than the original file. The file will be uploaded using gzip.
[/build]   ??  ::debug::The gzip file created for /tmp/forgejo-upload/forgejo-7.0-dev is smaller than the original file. The file will be uploaded using gzip.
[/build]   | ::debug::The gzip file created for /tmp/forgejo-upload/forgejo-7.0-dev is smaller than the original file. The file will be uploaded using gzip.
[/build]   | Total file count: 2 ---- Processed file #0 (0.0%)
[/build]   | Total file count: 2 ---- Processed file #0 (0.0%)
[/build]   | Total file count: 2 ---- Processed file #0 (0.0%)
[/build]   | Total file count: 2 ---- Processed file #0 (0.0%)
[/build]   | Total file count: 2 ---- Processed file #0 (0.0%)
[/build]   | Total file count: 2 ---- Processed file #0 (0.0%)
[/build]   | Total file count: 2 ---- Processed file #0 (0.0%)
[/build]   | Total file count: 2 ---- Processed file #0 (0.0%)
[/build]   | Total file count: 2 ---- Processed file #0 (0.0%)
[/build]   | Total file count: 2 ---- Processed file #0 (0.0%)
[/build]   | Total file count: 2 ---- Processed file #0 (0.0%)
[/build]   | Total file count: 2 ---- Processed file #0 (0.0%)
...
[/build]   | Total file count: 2 ---- Processed file #0 (0.0%)
[/build]   | Total file count: 2 ---- Processed file #0 (0.0%)
[/build]   | Total file count: 2 ---- Processed file #0 (0.0%)
[/build]   | Total file count: 2 ---- Processed file #0 (0.0%)
[/build]   | An error has been caught http-client index 0, retrying the upload
[/build]   | Error: socket hang up
[/build]   |     at connResetException (node:internal/errors:721:14)
[/build]   |     at TLSSocket.socketOnEnd (node:_http_client:519:23)
[/build]   |     at TLSSocket.emit (node:events:526:35)
[/build]   |     at endReadableNT (node:internal/streams/readable:1408:12)
[/build]   |     at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
[/build]   |   code: 'ECONNRESET'
[/build]   | }
[/build]   | Exponential backoff for retry #1. Waiting for 6361 milliseconds before continuing the upload at offset 0
[/build]   | Total file count: 2 ---- Processed file #0 (0.0%)
[/build]   | Finished backoff for retry #1, continuing with upload
[/build]   | Total file count: 2 ---- Processed file #0 (0.0%)
[/build]   | Total file count: 2 ---- Processed file #0 (0.0%)

Server

2024/04/17 20:23:33 ...eb/routing/logger.go:102:func1() [I] router: completed PUT /api/actions_pipeline/_apis/pipelines/workflows/16642/artifacts/6eff8731602e12a72be3802a16f1d0c9/upload?itemPath=forgejo-dev%2Fforgejo-7.0-dev for 144.76.140.197:0, 200 OK in 17.1ms @ <autogenerated>:1(actions.artifactRoutes.uploadArtifact-fm)

Screenshots

No response

Git Version

No response

Operating System

No response

How are you running Forgejo?

code.forgejo.org

Database

SQLite

### Description Re-running a workflow for which an artifact was already successfully uploaded failed with the upload action logging the following. There were no error messages server side. The same workflow in the same instance successfully uploaded artifacts an hour before. New runs of the same workflow now permanently fail to upload artifacts with the same behavior. ### Forgejo Version 7.0.0-dev-2136-f075579c95+gitea-1.22.0 ### Can you reproduce the bug on Forgejo Next? No ### Logs ``` [/build] | ::debug::Root artifact directory is /tmp/forgejo-upload [/build] | Starting artifact upload [/build] | For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-s\ tep-debug-logging Artifact name is valid! [/build] ?? ::debug::Artifact Url: https://code.forgejo.org/api/actions_pipeline/_apis/pipelines/workflows/16640/artifacts?api-version=6.0-preview [/build] | ::debug::Artifact Url: https://code.forgejo.org/api/actions_pipeline/_apis/pipelines/workflows/16640/artifacts?api-version=6.0-preview [/build] ?? ::debug::Upload Resource URL: https://code.forgejo.org/api/actions_pipeline/_apis/pipelines/workflows/16640/artifacts/6eff8731602e12a72be3802a16f1d0c9/upload [/build] | ::debug::Upload Resource URL: https://code.forgejo.org/api/actions_pipeline/_apis/pipelines/workflows/16640/artifacts/6eff8731602e12a72be3802a16f1d0c9/upload | ExponentiContainer for artifact "forgejo-dev" successfully created. Starting upload of file(s) [/build] ?? ::debug::File Concurrency: 2, and Chunk Size: 8388608 [/build] | ::debug::File Concurrency: 2, and Chunk Size: 8388608 [/build] ?? ::debug::/tmp/forgejo-upload/forgejo-7.0-dev is greater than 64k in size. Creating a gzip file on-disk /tmp/tmp-21867-AS36ytViavti to potentially reduce the upload size [/build] | ::debug::/tmp/forgejo-upload/forgejo-7.0-dev is greater than 64k in size. Creating a gzip file on-disk /tmp/tmp-21867-AS36ytViavti to potentially reduce the upload size [/build] ?? ::debug::/tmp/forgejo-upload/forgejo-8.0-dev is greater than 64k in size. Creating a gzip file on-disk /tmp/tmp-21867-YvEbmfwK2uho to potentially reduce the upload size [/build] | ::debug::/tmp/forgejo-upload/forgejo-8.0-dev is greater than 64k in size. Creating a gzip file on-disk /tmp/tmp-21867-YvEbmfwK2uho to potentially reduce the upload size [/build] ?? ::debug::The gzip file created for /tmp/forgejo-upload/forgejo-8.0-dev is smaller than the original file. The file will be uploaded using gzip. [/build] | ::debug::The gzip file created for /tmp/forgejo-upload/forgejo-8.0-dev is smaller than the original file. The file will be uploaded using gzip. [/build] ?? ::debug::The gzip file created for /tmp/forgejo-upload/forgejo-7.0-dev is smaller than the original file. The file will be uploaded using gzip. [/build] | ::debug::The gzip file created for /tmp/forgejo-upload/forgejo-7.0-dev is smaller than the original file. The file will be uploaded using gzip. [/build] | Total file count: 2 ---- Processed file #0 (0.0%) [/build] | Total file count: 2 ---- Processed file #0 (0.0%) [/build] | Total file count: 2 ---- Processed file #0 (0.0%) [/build] | Total file count: 2 ---- Processed file #0 (0.0%) [/build] | Total file count: 2 ---- Processed file #0 (0.0%) [/build] | Total file count: 2 ---- Processed file #0 (0.0%) [/build] | Total file count: 2 ---- Processed file #0 (0.0%) [/build] | Total file count: 2 ---- Processed file #0 (0.0%) [/build] | Total file count: 2 ---- Processed file #0 (0.0%) [/build] | Total file count: 2 ---- Processed file #0 (0.0%) [/build] | Total file count: 2 ---- Processed file #0 (0.0%) [/build] | Total file count: 2 ---- Processed file #0 (0.0%) ... [/build] | Total file count: 2 ---- Processed file #0 (0.0%) [/build] | Total file count: 2 ---- Processed file #0 (0.0%) [/build] | Total file count: 2 ---- Processed file #0 (0.0%) [/build] | Total file count: 2 ---- Processed file #0 (0.0%) [/build] | An error has been caught http-client index 0, retrying the upload [/build] | Error: socket hang up [/build] | at connResetException (node:internal/errors:721:14) [/build] | at TLSSocket.socketOnEnd (node:_http_client:519:23) [/build] | at TLSSocket.emit (node:events:526:35) [/build] | at endReadableNT (node:internal/streams/readable:1408:12) [/build] | at process.processTicksAndRejections (node:internal/process/task_queues:82:21) { [/build] | code: 'ECONNRESET' [/build] | } [/build] | Exponential backoff for retry #1. Waiting for 6361 milliseconds before continuing the upload at offset 0 [/build] | Total file count: 2 ---- Processed file #0 (0.0%) [/build] | Finished backoff for retry #1, continuing with upload [/build] | Total file count: 2 ---- Processed file #0 (0.0%) [/build] | Total file count: 2 ---- Processed file #0 (0.0%) ``` Server ``` 2024/04/17 20:23:33 ...eb/routing/logger.go:102:func1() [I] router: completed PUT /api/actions_pipeline/_apis/pipelines/workflows/16642/artifacts/6eff8731602e12a72be3802a16f1d0c9/upload?itemPath=forgejo-dev%2Fforgejo-7.0-dev for 144.76.140.197:0, 200 OK in 17.1ms @ <autogenerated>:1(actions.artifactRoutes.uploadArtifact-fm) ``` ### Screenshots _No response_ ### Git Version _No response_ ### Operating System _No response_ ### How are you running Forgejo? code.forgejo.org ### Database SQLite

earl-warren added the

Kind/Bug

label 2024-04-17 21:48:57 +00:00

earl-warren commented

2024-04-17 21:49:23 +00:00

Author

Owner

This is the log of a sucessful upload that happened earlier the same day.

[/build]   | ::debug::Upload Resource URL: https://code.forgejo.org/api/actions_pipeline/_apis/pipelines/workflows/16632/artifacts/6eff8731602e12a72be3802a16f1d0c9/upload
[/build]   | Container for artifact "forgejo-dev" successfully created. Starting upload of file(s)
[/build]   <F0><9F><92><AC>  ::debug::File Concurrency: 2, and Chunk Size: 8388608
[/build]   | ::debug::File Concurrency: 2, and Chunk Size: 8388608
[/build]   <F0><9F><92><AC>  ::debug::/tmp/forgejo-upload/forgejo-7.0-dev is greater than 64k in size. Creating a gzip file on-disk /tmp/tmp-21911-3SHJjlfpjnwc to potentially reduce the upload size
[/build]   | ::debug::/tmp/forgejo-upload/forgejo-7.0-dev is greater than 64k in size. Creating a gzip file on-disk /tmp/tmp-21911-3SHJjlfpjnwc to potentially reduce the upload size
[/build]   <F0><9F><92><AC>  ::debug::/tmp/forgejo-upload/forgejo-8.0-dev is greater than 64k in size. Creating a gzip file on-disk /tmp/tmp-21911-gNSTZBcVrd71 to potentially reduce the upload size
[/build]   | ::debug::/tmp/forgejo-upload/forgejo-8.0-dev is greater than 64k in size. Creating a gzip file on-disk /tmp/tmp-21911-gNSTZBcVrd71 to potentially reduce the upload size
[/build]   <F0><9F><92><AC>  ::debug::The gzip file created for /tmp/forgejo-upload/forgejo-8.0-dev is smaller than the original file. The file will be uploaded using gzip.
[/build]   | ::debug::The gzip file created for /tmp/forgejo-upload/forgejo-8.0-dev is smaller than the original file. The file will be uploaded using gzip.
[/build]   <F0><9F><92><AC>  ::debug::The gzip file created for /tmp/forgejo-upload/forgejo-7.0-dev is smaller than the original file. The file will be uploaded using gzip.
[/build]   | ::debug::The gzip file created for /tmp/forgejo-upload/forgejo-7.0-dev is smaller than the original file. The file will be uploaded using gzip.
[/build]   | Uploaded /tmp/forgejo-upload/forgejo-8.0-dev (24.1%) bytes 0:8388607
[/build]   | Uploaded /tmp/forgejo-upload/forgejo-7.0-dev (23.9%) bytes 0:8388607
[/build]   | Uploaded /tmp/forgejo-upload/forgejo-8.0-dev (48.3%) bytes 8388608:16777215
[/build]   | Uploaded /tmp/forgejo-upload/forgejo-7.0-dev (47.8%) bytes 8388608:16777215
[/build]   | Uploaded /tmp/forgejo-upload/forgejo-8.0-dev (72.5%) bytes 16777216:25165823
[/build]   | Uploaded /tmp/forgejo-upload/forgejo-7.0-dev (71.7%) bytes 16777216:25165823
[/build]   | Uploaded /tmp/forgejo-upload/forgejo-8.0-dev (96.7%) bytes 25165824:33554431
[/build]   | Uploaded /tmp/forgejo-upload/forgejo-8.0-dev (100.0%) bytes 33554432:34692846
[/build]   <F0><9F><92><AC>  ::debug::deleting temporary gzip file /tmp/tmp-21911-gNSTZBcVrd71
[/build]   | ::debug::deleting temporary gzip file /tmp/tmp-21911-gNSTZBcVrd71
[/build]   | Uploaded /tmp/forgejo-upload/forgejo-7.0-dev (95.6%) bytes 25165824:33554431
[/build]   | Uploaded /tmp/forgejo-upload/forgejo-7.0-dev (100.0%) bytes 33554432:35064996
[/build]   <F0><9F><92><AC>  ::debug::deleting temporary gzip file /tmp/tmp-21911-3SHJjlfpjnwc
[/build]   | ::debug::deleting temporary gzip file /tmp/tmp-21911-3SHJjlfpjnwc
[/build]   | Total size of all the files uploaded is 69757844 bytes
[/build]   | File upload process has finished. Finalizing the artifact upload
[/build]   <F0><9F><92><AC>  ::debug::Artifact Url: https://code.forgejo.org/api/actions_pipeline/_apis/pipelines/workflows/16632/artifacts?api-version=6.0-preview
[/build]   | ::debug::Artifact Url: https://code.forgejo.org/api/actions_pipeline/_apis/pipelines/workflows/16632/artifacts?api-version=6.0-preview
[/build]   <F0><9F><92><AC>  ::debug::URL is https://code.forgejo.org/api/actions_pipeline/_apis/pipelines/workflows/16632/artifacts?api-version=6.0-preview&artifactName=forgejo-dev
[/build]   | ::debug::URL is https://code.forgejo.org/api/actions_pipeline/_apis/pipelines/workflows/16632/artifacts?api-version=6.0-preview&artifactName=forgejo-dev
[/build]   <F0><9F><92><AC>  ::debug::Artifact forgejo-dev has been successfully uploaded, total size in bytes: 197074752
[/build]   | ::debug::Artifact forgejo-dev has been successfully uploaded, total size in bytes: 197074752
[/build]   | Artifact has been finalized. All files have been successfully uploaded!
[/build]   | 
[/build]   | The raw size of all the files that were specified for upload is 197074752 bytes
[/build]   | The size of all the files that were uploaded is 69757844 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage
[/build]   | 
[/build]   | Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 
[/build]   | 
[/build]   | Artifact forgejo-dev has been successfully uploaded!
[/build]   <E2><9C><85>  Success - Main actions/upload-artifact@v3

This is the log of a sucessful upload that happened earlier the same day. ``` [/build] | ::debug::Upload Resource URL: https://code.forgejo.org/api/actions_pipeline/_apis/pipelines/workflows/16632/artifacts/6eff8731602e12a72be3802a16f1d0c9/upload [/build] | Container for artifact "forgejo-dev" successfully created. Starting upload of file(s) [/build] <F0><9F><92><AC> ::debug::File Concurrency: 2, and Chunk Size: 8388608 [/build] | ::debug::File Concurrency: 2, and Chunk Size: 8388608 [/build] <F0><9F><92><AC> ::debug::/tmp/forgejo-upload/forgejo-7.0-dev is greater than 64k in size. Creating a gzip file on-disk /tmp/tmp-21911-3SHJjlfpjnwc to potentially reduce the upload size [/build] | ::debug::/tmp/forgejo-upload/forgejo-7.0-dev is greater than 64k in size. Creating a gzip file on-disk /tmp/tmp-21911-3SHJjlfpjnwc to potentially reduce the upload size [/build] <F0><9F><92><AC> ::debug::/tmp/forgejo-upload/forgejo-8.0-dev is greater than 64k in size. Creating a gzip file on-disk /tmp/tmp-21911-gNSTZBcVrd71 to potentially reduce the upload size [/build] | ::debug::/tmp/forgejo-upload/forgejo-8.0-dev is greater than 64k in size. Creating a gzip file on-disk /tmp/tmp-21911-gNSTZBcVrd71 to potentially reduce the upload size [/build] <F0><9F><92><AC> ::debug::The gzip file created for /tmp/forgejo-upload/forgejo-8.0-dev is smaller than the original file. The file will be uploaded using gzip. [/build] | ::debug::The gzip file created for /tmp/forgejo-upload/forgejo-8.0-dev is smaller than the original file. The file will be uploaded using gzip. [/build] <F0><9F><92><AC> ::debug::The gzip file created for /tmp/forgejo-upload/forgejo-7.0-dev is smaller than the original file. The file will be uploaded using gzip. [/build] | ::debug::The gzip file created for /tmp/forgejo-upload/forgejo-7.0-dev is smaller than the original file. The file will be uploaded using gzip. [/build] | Uploaded /tmp/forgejo-upload/forgejo-8.0-dev (24.1%) bytes 0:8388607 [/build] | Uploaded /tmp/forgejo-upload/forgejo-7.0-dev (23.9%) bytes 0:8388607 [/build] | Uploaded /tmp/forgejo-upload/forgejo-8.0-dev (48.3%) bytes 8388608:16777215 [/build] | Uploaded /tmp/forgejo-upload/forgejo-7.0-dev (47.8%) bytes 8388608:16777215 [/build] | Uploaded /tmp/forgejo-upload/forgejo-8.0-dev (72.5%) bytes 16777216:25165823 [/build] | Uploaded /tmp/forgejo-upload/forgejo-7.0-dev (71.7%) bytes 16777216:25165823 [/build] | Uploaded /tmp/forgejo-upload/forgejo-8.0-dev (96.7%) bytes 25165824:33554431 [/build] | Uploaded /tmp/forgejo-upload/forgejo-8.0-dev (100.0%) bytes 33554432:34692846 [/build] <F0><9F><92><AC> ::debug::deleting temporary gzip file /tmp/tmp-21911-gNSTZBcVrd71 [/build] | ::debug::deleting temporary gzip file /tmp/tmp-21911-gNSTZBcVrd71 [/build] | Uploaded /tmp/forgejo-upload/forgejo-7.0-dev (95.6%) bytes 25165824:33554431 [/build] | Uploaded /tmp/forgejo-upload/forgejo-7.0-dev (100.0%) bytes 33554432:35064996 [/build] <F0><9F><92><AC> ::debug::deleting temporary gzip file /tmp/tmp-21911-3SHJjlfpjnwc [/build] | ::debug::deleting temporary gzip file /tmp/tmp-21911-3SHJjlfpjnwc [/build] | Total size of all the files uploaded is 69757844 bytes [/build] | File upload process has finished. Finalizing the artifact upload [/build] <F0><9F><92><AC> ::debug::Artifact Url: https://code.forgejo.org/api/actions_pipeline/_apis/pipelines/workflows/16632/artifacts?api-version=6.0-preview [/build] | ::debug::Artifact Url: https://code.forgejo.org/api/actions_pipeline/_apis/pipelines/workflows/16632/artifacts?api-version=6.0-preview [/build] <F0><9F><92><AC> ::debug::URL is https://code.forgejo.org/api/actions_pipeline/_apis/pipelines/workflows/16632/artifacts?api-version=6.0-preview&artifactName=forgejo-dev [/build] | ::debug::URL is https://code.forgejo.org/api/actions_pipeline/_apis/pipelines/workflows/16632/artifacts?api-version=6.0-preview&artifactName=forgejo-dev [/build] <F0><9F><92><AC> ::debug::Artifact forgejo-dev has been successfully uploaded, total size in bytes: 197074752 [/build] | ::debug::Artifact forgejo-dev has been successfully uploaded, total size in bytes: 197074752 [/build] | Artifact has been finalized. All files have been successfully uploaded! [/build] | [/build] | The raw size of all the files that were specified for upload is 197074752 bytes [/build] | The size of all the files that were uploaded is 69757844 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage [/build] | [/build] | Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads [/build] | [/build] | Artifact forgejo-dev has been successfully uploaded! [/build] <E2><9C><85> Success - Main actions/upload-artifact@v3 ```

earl-warren commented

2024-04-17 21:50:01 +00:00

Author

Owner

Verified both artifacts v3 & v4 are still working fine on another repo.

Verified both artifacts v3 & v4 are still working fine on another repo. * https://code.forgejo.org/forgejo/test-ignore/actions/runs/3 * https://code.forgejo.org/forgejo/test-ignore/actions/runs/2

earl-warren commented

2024-04-17 21:52:38 +00:00

Author

Owner

The problem went away after removing the runner cache (150GB big). No backup was kept for forensic analysis unfortunately. When it happens again the content of the cache for the repository that is being stuck this way will help. It blocks all artifact v3 uploads for this repository but does not impact the others.