fix: implement idempotent FetchTask API calls to reduce risk of lost tasks #1393

Merged

mfenniak merged 1 commit from mfenniak/forgejo-runner:fetchtask-idempotent into main 2026-02-22 22:24:44 +00:00

Conversation 2 Commits 1 Files changed 8 +237 -35

mfenniak commented 2026-02-20 17:32:04 +00:00

Owner

Copy link

When the FetchTask() API is invoked to create a task, unpreventable environmental errors may occur; for example, network disconnects and timeouts. It's possible that these errors occur after the server-side has assigned a task to the runner during the API call, in which case the error would cause that task to be lost between the two systems -- the server will think it's assigned to the runner, and the runner never received it. This can cause jobs to appear stuck at "Set up job" (#1391).

The solution implemented here is idempotency in the FetchTask() API call, which means that the "same" FetchTask() API call is expected to return the same values. Specifically, the runner creates a unique identifier requestKey which is transmitted to the server along with each FetchTask() invocation which defines the sameness of the call, and the runner retains the requestKey value until the API call receives a successful response. If the server implements idempotency, it can use this key to identify repeated invocations of FetchTask() and when the same request is received, the same response is provided.

Runner's responsibility is to send the same request key consistently if any error occurred, and, change it to a new key when a successful call is received.

A separate Forgejo PR will be coming to implement the server-side portion of this, and I expect to hold this PR until the server-side is reviewed to ensure the system design doesn't change during that review and invalidate any work here.

• bug fixes
  • PR: fix: implement idempotent FetchTask API calls to reduce risk of lost tasks

When the FetchTask() API is invoked to create a task, unpreventable environmental errors may occur; for example, network disconnects and timeouts. It's possible that these errors occur after the server-side has assigned a task to the runner during the API call, in which case the error would cause that task to be lost between the two systems -- the server will think it's assigned to the runner, and the runner never received it. This can cause jobs to appear stuck at "Set up job" (#1391). The solution implemented here is idempotency in the FetchTask() API call, which means that the "same" FetchTask() API call is expected to return the same values. Specifically, the runner creates a unique identifier `requestKey` which is transmitted to the server along with each FetchTask() invocation which defines the sameness of the call, and the runner retains the `requestKey` value until the API call receives a successful response. If the server implements idempotency, it can use this key to identify repeated invocations of FetchTask() and when the same request is received, the same response is provided. Runner's responsibility is to send the same request key consistently if any error occurred, and, change it to a new key when a successful call is received. A separate Forgejo PR will be coming to implement the server-side portion of this, and I expect to hold this PR until the server-side is reviewed to ensure the system design doesn't change during that review and invalidate any work here. <!--start release-notes-assistant--> <!--URL:https://code.forgejo.org/forgejo/runner--> - bug fixes - [PR](https://code.forgejo.org/forgejo/runner/pulls/1393): <!--number 1393 --><!--line 0 --><!--description Zml4OiBpbXBsZW1lbnQgaWRlbXBvdGVudCBGZXRjaFRhc2sgQVBJIGNhbGxzIHRvIHJlZHVjZSByaXNrIG9mIGxvc3QgdGFza3M=-->fix: implement idempotent FetchTask API calls to reduce risk of lost tasks<!--description--> <!--end release-notes-assistant-->

mfenniak force-pushed fetchtask-idempotent from cf9dba7ba5 to 7b3facef73 2026-02-20 17:33:28 +00:00 Compare

mfenniak force-pushed fetchtask-idempotent from 7b3facef73 to a57a4664e3 2026-02-20 18:28:10 +00:00 Compare

aahlenst approved these changes 2026-02-20 20:51:51 +00:00

mfenniak added the

run-end-to-end-tests

label 2026-02-22 02:46:25 +00:00

cascading-pr referenced this pull request from actions/setup-forgejo 2026-02-22 02:46:37 +00:00

cascading-pr from https://code.forgejo.org/forgejo/runner refs/pull/1393/head to forgejo/runner-1393 #903

cascading-pr commented 2026-02-22 02:46:38 +00:00

Contributor

Copy link

cascading-pr updated at actions/setup-forgejo#903

cascading-pr updated at https://code.forgejo.org/actions/setup-forgejo/pulls/903

mfenniak merged commit 142689a39a into main 2026-02-22 22:24:44 +00:00

mfenniak deleted branch fetchtask-idempotent 2026-02-22 22:24:44 +00:00

mfenniak referenced this pull request from a commit 2026-02-22 22:24:44 +00:00

fix: implement idempotent FetchTask API calls to reduce risk of lost tasks (#1393)

mfenniak referenced this pull request 2026-02-22 22:34:18 +00:00

bug: jobs stuck at "Set up job" #1391

forgejo-mirror referenced this pull request from a commit 2026-02-22 23:01:33 +00:00

fix: allow Actions runner to recover tasks lost during fetching from intermittent errors (#11401)

viceice-bot referenced this pull request from actions/setup-forgejo 2026-02-28 00:02:24 +00:00

Update dependency forgejo/runner to v12.7.1 #908

viceice-bot referenced this pull request 2026-02-28 00:13:24 +00:00

Update forgejo-runner to v12.7.1 #1415

mfenniak referenced this pull request from a commit 2026-02-28 15:13:54 +00:00

Update forgejo-runner to v12.7.1 (#1415)

mfenniak referenced this pull request from a commit 2026-02-28 15:14:33 +00:00

Update dependency forgejo/runner to v12.7.1 (#908)

forgejo-mirror referenced this pull request from a commit 2026-03-08 02:01:46 +00:00

Update module code.forgejo.org/forgejo/runner/v12 to v12.7.1 (forgejo) (#11562)

albundy83 referenced this pull request from a commit 2026-03-17 22:18:39 +00:00

[kubernetes] Add example to use buildkit and docker buildx

albundy83 referenced this pull request from a commit 2026-03-17 22:27:05 +00:00

[kubernetes] Add example to use buildkit and docker buildx

mfenniak referenced this pull request 2026-03-26 16:09:45 +00:00

fix: increase log level of client-side fetch timeout #1452

mfenniak referenced this pull request from a commit 2026-03-26 18:35:36 +00:00

fix: increase log level of client-side fetch timeout (#1452)

mfenniak referenced this pull request 2026-03-27 16:46:23 +00:00

bug: runner can exceed configured runner.capacity w/ multiple Forgejo connections and idempotent requests #1453

Sign in to join this conversation.

Reviewers

No reviewers

aahlenst

Labels

Clear labels   

FreeBSD

  

Kind/Breaking

Breaking change that won't be backward compatible

  

Kind/Bug

Something is not working

  

Kind/Chore

It is no fun but it needs to be done, this is a chore.

  

Kind/DependencyUpdate

for pull requests that are created by renovate when updating a dependency

  

Kind/Documentation

Documentation changes

  

Kind/Enhancement

Improve existing functionality

  

Kind/Feature

New functionality

  

Kind/Security

This is security issue

  

Kind/Testing

Issue or pull request related to testing

  

Priority

Critical

The priority is critical

  

Priority

High

The priority is high

  

Priority

Low

The priority is low

  

Priority

Medium

The priority is medium

  

Reviewed

Confirmed

Issue has been confirmed

  

Reviewed

Duplicate

This issue or pull request already exists

  

Reviewed

Invalid

Invalid issue

  

Reviewed

Won't Fix

This issue won't be fixed

  

Status

Abandoned

Somebody has started to work on this but abandoned work

  

Status

Blocked

Something is blocking this issue or pull request

  

Status

Need More Info

Feedback is required to reproduce issue or to continue work

  

Windows

Windows ports is an independent project, see https://github.com/Crown0815/Forgejo-runner-windows-builder

  

linux-powerpc64le

powerpc64le

  

linux-riscv64

  

linux-s390x

  

run-end-to-end-tests

Run https://code.forgejo.org/forgejo/end-to-end with a runner compiled from this pull request

  

run-forgejo-tests

Run https://codeberg.org/forgejo/forgejo tests importing the runner from this pull request

No labels

FreeBSD

Kind/Breaking

Kind/Bug

Kind/Chore

Kind/DependencyUpdate

Kind/Documentation

Kind/Enhancement

Kind/Feature

Kind/Security

Kind/Testing

Priority

Critical

Priority

High

Priority

Low

Priority

Medium

Reviewed

Confirmed

Reviewed

Duplicate

Reviewed

Invalid

Reviewed

Won't Fix

Status

Abandoned

Status

Blocked

Status

Need More Info

Windows

linux-powerpc64le

linux-riscv64

linux-s390x

run-end-to-end-tests

run-forgejo-tests

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

3 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

forgejo/runner!1393

No description provided.