gof3

Gof3 is a Go package and a CLI that provides a single operation: mirroring. The origin and destination are designated by the URL of a forge and a path to the resource. For instance, mirror --from-type forgejo --from https://code.forgejo.org/f3/f3-cli-example --to-type filesystem --to /some/directory will mirror the f3-cli-example project in a local directory using the F3 format.

Building

• Install go >= v1.26
• make f3-cli
• ./f3-cli mirror -h

Example

To F3

Login to https://code.forgejo.org and obtain an application token with read permissions at https://code.forgejo.org/user/settings/applications.

f3-cli mirror \
  --from-type forgejo --from-forgejo-url https://code.forgejo.org \
  --from-forgejo-token $codetoken \
  --from-path /forge/organizations/f3/projects/f3-cli-example \
  --to-type filesystem --to-filesystem-directory /tmp/f3-cli-example

From F3

Run a local Forgejo instance with ./tests/run.sh run_forgejo and get the application token from the content of the /tmp/forgejo-end-to-end/forgejo-curl/token file.

localtoken=$(cat /tmp/forgejo-end-to-end/forgejo-curl/token)
f3-cli mirror \
  --from-type filesystem --from-filesystem-directory /tmp/f3-cli-example \
  --from-path /forge/organizations/f3/projects/f3-cli-example \
  --to-type forgejo --to-forgejo-url http://0.0.0.0:3001 \
  --to-forgejo-token $localtoken

Visit the project that was just created at http://0.0.0.0:3001/f3/f3-cli-example http://0.0.0.0:3001/f3/f3-cli-example

License

This project is MIT licensed.

Architecture

F3 is a hierarchy designed to be stored in a file system. It is represented in memory with the tree/generic abstract data structure that can be saved and loaded from disk by the forges/filesystem driver. Each forge (e.g. forges/forgejo) is supported by a driver that is responsible for the interactions of each resource (e.g issues, asset, etc.).

Tree

tree/f3 implements a F3 hierarchy based on the tree/generic data structure. The tree has a logger for messages, options defining which forge it relates to and how and a pointer to the root node of the hierarchy (i.e. the forge F3 resource).

The node (tree/generic/node.go) has:

• a unique id (e.g. the numerical id of an issue)
• a parent
• chidren (e.g. issues children are issues, issue children are comments and reactions)
• a kind that maps to a F3 resource (e.g. issue, etc.)
• a driver for its concrete implementation for a given forge

The node relies on a forge driver for the concrete implemenation of a F3 resource (issue, reaction, repository, etc.). For instance the issues driver for Forgejo is responsible for listing the existing issues and the issue driver is responsible for creating, updating or deleting a Forgejo issue.

F3 archive

The F3 JSON schemas are copied in f3/schemas. Their internal representation and validation is found in a source file named after the resource (e.g. an issue represented by f3/schemas/issue.json is implemented by f3/issue.go).

When a F3 resource includes data external to the JSON file (i.e. a Git repository or an asset file), the internal representation has a function to copy the data to the destination given in argument. For instance:

• f3/repository.go FetchFunc(destination ...) will git fetch --mirror the repository to the destination directory.
• f3/attachment.go DownloadFunc() returns a io.ReadCloser that will be used by the caller to copy the asset to its destination.

Options

The Forge options at options/interface.go define the parameters given when a forge is created.

Each forge driver is responsible for registering the options (e.g. Forgejo options) and for registering a factory that will create these options (e.g. Forgejo options registration). In addition to the options that are shared by all forges such as the logger, it may define additional options.

Driver interface

For each F3 resource, the driver implements an interface.

• FromFormat is given the f3 resource and converts it into an internal representation
• ToFormat does the opposite

The driver operations will then be used to:

• Get the resource from the forge and stor it in the internal representation
• Put the internal representation to the forge when it does not already exist
• Patch the forge resource with the modifications of an existing resource using the internal representation
• Delete a that is known to exist in the forge

A driver must have a unique name (e.g. forgejo) and register (e.g. Forgejo registration):

• an options factory
• a forge factory

Tree driver

The tree driver functions (e.g. forges/forgejo/tree.go) specialize NullTreeDriver.

• Factory(ctx context.Context, kind generic.Kind) generic.NodeDriverInterface creates a new node driver for a given Kind.
• GetPageSize() int returns the default page size.

Node driver

The node driver functions for each Kind (e.g. issues, issue, etc.) specialize NullNodeDriver. The examples are given for the Forgejo issue and issues drivers, matching the REST API endpoint to the driver function.

• ListPage(context.Context, page int) ChildrenSlice returns children of the node paginated (e.g. GET /repos/{owner}/{repo}/issues)
• Get(context.Context) get the content of the resource (e.g. GET /repos/{owner}/{repo}/issues/{index})
• Put(context.Context) NodeID create a new resource and return the identifier (e.g. POST /repos/{owner}/{repo}/issues)
• Patch(context.Context) modify an existing resource (e.g. PATCH /repos/{owner}/{repo}/issues/{index})
• Delete(context.Context) delete an existing resource (e.g. DELETE /repos/{owner}/{repo}/issues/{index})
• NewFormat() f3.Interface create a new issue F3 object
• FromFormat(f3.Interface) set the internal representation from the given F3 resource
• ToFormat() f3.Interface convert the internal representation into the corresponding F3 resource. (e.g. the internal representation of an issue for the Forgejo driver is the Issue struct of the Forgejo SDK).

Options

The options created by the factory are expected to provide the options interfaces:

• Required
  • LoggerInterface
  • URLInterface
• Optional
  • CLIInterface if additional CLI arguments specific to the forge are supported
  • AuthInterface if accessing the forge requires authentication
  • HTTPInterface if the forge is providing a REST API (e.g. supporting proxy, obeying rate-limits, ...)

For instance forges/forgejo/options/options.go is created by forges/forgejo/options.go.

Driver implementation

A driver for a forge must be self contained in a directory (e.g. forges/forgejo). Functions shared by multiple forges are grouped in the forges/helpers directory and split into one directory per Kind (e.g. forges/helpers/pullrequest).

• options.go defines the name of the forge in the Name variable (e.g. Name = "forgejo")
• options/options.go defines the options specific to the forge and the corresponding CLI flags
• main.go calls f3_tree.RegisterForgeFactory to create the forge given its name
• tree.go has the Factory() function that maps a node kind (issue, reaction, etc.) into an object that is capable of interacting with it (CRUD).
• one file per Kind (e.g. forges/forgejo/issues.go).

Idempotency

Mirroring is idempotent: it will produce the same result if repeated multiple times. The drivers functions are not required to be idempotent.

• The Put function will only be called if the resource does not already exist.
• The Patch and Delete functions will only be called if the resource exists.

Identifiers mapping

When a forge (e.g. Forgejo) is mirrored to the filesystem, the identifiers are preserved verbatim (e.g. the issue identifier). When the filesystem is mirrored to a forge, the identifiers cannot always be preserved. For instance if an issue with the identifier 1234 is downloaded from Forgejo and copied on another Forgejo instance, it will be allocated an identifier by the Forgejo instance. It cannot request to be given the 1234 identifier.

References

A F3 resource may reference another F3 resource by a path. For instance the user that authored an issue is represented by /forge/users/1234 where 1234 is the unique identifier of the user. The reference is relative to the forge. The mirroring of a forge to another is responsible for converting the references using the identifier mapping stored in the origin forge. For instance if /forge/users/1234 stored in the filesystem is mirrored in Forgejo as /forge/users/58, an issue authored by /forge/users/1234 in the filesystem will be mirrored in Forgejo as an issue authored by /forge/users/58.

Logger

The tree/generic has a pointer to a logger implementing logger.Interface which is made available to the nodes and the drivers.

Context

All functions except for setters and getters have a context.Context argument which is checked (using util/terminate.go) to not be Done before performing a blocking operation (e.g. a REST API call or a call to the Git CLI).

The context is not meant to store data, even temporarily.

Error model

No recoverable errors panic. If the error is recoverable, an error is returned to the caller.

CLI

The CLI is in cmd and relies on options to figure out which options are to be implemented for each supported forge.

Contributor guide

Requirements

The tests will use up to 16GB of RAM (mainly because it creates live GitLab instance).

• A Debian/Ubuntu based GNU/Linux OS
• Install docker

What to work on

For your first contribution pick something easy.

Add test coverage (easy)

• Follow the instructions in the Code coverage section below to find functions that do not have 100% coverage
• Find which test already covers the function and improve it to cover the missing lines

Add a new test (medium difficulty)

• Follow the instructions in the Code coverage section below to find functions that have no test coverage at all
• Write a new test to at least partially test this function

Add a new resource driver (medium difficulty)

• Add a new resource driver in a forges subdirectory (e.g. if forges/gitlab/reactions.go does not exist, there is a need for a new resource driver)
• Use the resource driver from forges/forgejo as a reference implementation

Improve the compliance suite (difficult)

• Add a new test case to tree/tests/f3/forge_compliance.go

AI requirements

When using an AI to author code, it is expected that AI interactions are traceable. You are required to:

• Archive the dialog you have with the AI in a chat/{username}-YYYY-MM-DD-HH:MM:SS file where username is your code.forgejo.org user name
• Commit the file before producing code
• Commit the code produced before resuming the dialog with the AI and starting another file

This will lead to a commit series that interleaves AI dialogs with code.

It helps the reviewer of the pull request when asking for changes to separate what you authored from the AI output. For instance, if the reviewer sees from what you authored that you are not going in the right direction, they may be inclined to spend time explaining what you have missed. If the same kind of misunderstanding shows in the AI output, they may ask you to send different prompts to the AI.

Local tests

The forge instance is deleted before each run and left running for forensic analysis when the run completes.

./tests/run.sh test_forgejo # http://0.0.0.0:3001 user root, password admin1234
./tests/run.sh test_gitlab # http://0.0.0.0:8181 user root, password Wrobyak4
./tests/run.sh test_gitea # http://0.0.0.0:3001 user root, password admin1234

Restart a new forge with:

./tests/run.sh run_forgejo # http://0.0.0.0:3001 user root, password admin1234
./tests/run.sh run_gitlab # http://0.0.0.0:8181 user root, password Wrobyak4
./tests/run.sh run_gitea # http://0.0.0.0:3001 user root, password admin1234

The compliance test resources are deleted, except if the environment variable GOF3_TEST_COMPLIANCE_CLEANUP=false.

GOF3_TEST_COMPLIANCE_CLEANUP=false GOF3_FORGEJO_HOST_PORT=0.0.0.0:3001 go test -run=TestTree_F3_F3Forge/forgejo -v code.forgejo.org/f3/gof3/...

Code coverage

All

export SCRATCHDIR=/tmp/gof3 ; rm -fr /tmp/gof3
./tests/run.sh # will display function coverage when it completes
uncover /tmp/gof3/merged.out GeneratorSetReviewComment # show which lines of the GeneratorSetReviewComment function are not covered

All tests in a given package

rm -fr /tmp/gof3/* ; SCRATCHDIR=/tmp/gof3 ./tests/run.sh test_package tree  # will display function coverage when it completes
uncover /tmp/gof3/merged.out GeneratorSetReviewComment # show which lines of the GeneratorSetReviewComment function are not covered

One test found in a given package

rm -fr /tmp/gof3/* ; SCRATCHDIR=/tmp/gof3 ./tests/run.sh test_function tree/tests/f3 TestF3MirrorMarkdown # will display function coverage when it completes
uncover /tmp/gof3/merged.out BuildAndVerifyURL # show which lines of the BuildAndVerifyURL function are not covered

The filesystem forge

The tests are in a separate directory and assembling coverage does not fit in the above examples. It is covered when running tests on the tree/tests package but that also includes forgejo and takes longer, which may be inconvenient in the development loop.

rm -fr /tmp/gof3/* ; SCRATCHDIR=/tmp/gof3 ./tests/run.sh test_forge_filesystem

F3 schemas

The JSON schemas come from the f3-schemas repository and should be updated as follows:

cd f3 ; rm -fr schemas ; git --work-tree schemas clone https://code.forgejo.org/f3/f3-schemas ; rm -fr f3-schemas schemas/.gitignore schemas/.forgejo schemas/.editorconfig

Funding

See the page dedicated to funding in the F3 documentation