The `gitFiler` used by the FindLicense RPC hardcodes the SHA1 object
hash in order to parse the output returned by git-ls-tree(1). Fix this
and detect the object hash at runtime to fix compatibility with the
SHA256 object format.

Changelog: added

Convert non-service functionality to support SHA256

See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6102



Merged-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Approved-by: karthik nayak <knayak@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>

Allow including unreachable objects in transaction

Closes #5451

See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6095



Merged-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Reviewed-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Reviewed-by: Sami Hiltunen <shiltunen@gitlab.com>
Co-authored-by: Sami Hiltunen <shiltunen@gitlab.com>

proto: Introduce new UpdateReference RPC

Closes #5084

See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6037



Merged-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Approved-by: karthik nayak <knayak@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Reviewed-by: karthik nayak <knayak@gitlab.com>

proto: Add missing documentation for BlobService definitions

Closes #5113

See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6087



Merged-by: karthik nayak <knayak@gitlab.com>
Approved-by: karthik nayak <knayak@gitlab.com>
Reviewed-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Reviewed-by: Justin Tobler <jtobler@gitlab.com>
Co-authored-by: Patrick Steinhardt <psteinhardt@gitlab.com>

go: Update module github.com/pelletier/go-toml/v2 to v2.0.9

See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6096



Merged-by: karthik nayak <knayak@gitlab.com>
Approved-by: karthik nayak <knayak@gitlab.com>
Co-authored-by: GitLab Renovate Bot <gitlab-bot@gitlab.com>

We're using a regular expression to parse raw diff lines. This regular
expression has the length of SHA1 object hashes hardcoded and thus fails
to recognize SHA256 diff lines.

Plumb through the object hash used by the current repository so that we
can switch between either a SHA1- or SHA256-specific regexp. This makes
parsing of diffs work with both object formats.

Changelog: added

The diff tests use a bunch of hardcoded object IDs in the diffs we're
trying to parse, which are all hardcoded to SHA1 lengths. Refactor the
test to instead compute object IDs via the default object hash so that
they're dynamically adapted to both SHA1 and SHA256 lengths.

Especially in our tests we often need to generate some arbitrary test
object IDs. This is needlessly complex right now and either requires
some fiddling with `git.ObjectHash.EncodedLen()` or using the hasher
interface to compute the object ID.

Provide a new `HashData()` function that hashes data to either a SHA1 or
SHA256 object ID.

Convert tests for the gitaly-hooks command to be compatible with the
SHA256 object format by dropping pre-seeded repositories and not using
hardcoded object ID lengths.

The feature flags test for Git2go doesn't access any Git objects, so it
can be trivially enabled to test with SHA256.

Some RPCs in Gitaly, such as UserSquash, write objects into the repository
without making them reachable. The TransactionManager only includes objects
in the logged pack file that are reachable. This creates a problem for those
RPCs as the objects they attempt to write do not get committed into the
repository. This commit enables committing unreachable objects in a transaction.
The problematic RPCs can include specific objects in the pack even if they
are not reachable from the references so they get committed as part of the
transaction.

Some tests could benefit from testing with a commit that divereges
from the other history. Add such a commit to the test setup.

repository: Make `ReplicateRepository()` tests support SHA256

See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6053



Merged-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Approved-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Reviewed-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Co-authored-by: Justin Tobler <jtobler@gitlab.com>

go: Update module github.com/jackc/pgx/v5 to v5.4.2

See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6089



Merged-by: John Cai <jcai@gitlab.com>
Approved-by: John Cai <jcai@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Co-authored-by: GitLab Renovate Bot <gitlab-bot@gitlab.com>

.gitlab-ci.yml: Use environment to clean up branch

See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6065



Merged-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: Lin Jen-Shin <jen-shin@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Reviewed-by: Lin Jen-Shin <jen-shin@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Co-authored-by: John Cai <jcai@gitlab.com>

ci: Require macOS job to succeed again

See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6094



Merged-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Co-authored-by: Patrick Steinhardt <psteinhardt@gitlab.com>

tools/gotestsum: Update module gotest.tools/gotestsum to v1.10.1

See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6079



Merged-by: Christian Couder <chriscool@tuxfamily.org>
Approved-by: Christian Couder <chriscool@tuxfamily.org>
Approved-by: karthik nayak <knayak@gitlab.com>
Co-authored-by: GitLab Renovate Bot <gitlab-bot@gitlab.com>

Seeded repositories are not supported with SHA256 object formats.
Replace the use of seeded repositories for `ReplicateRepository()` RPC
tests with repositories generated at test execution.

go: Update github.com/ProtonMail/go-crypto digest to e01326f

See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6051



Merged-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Approved-by: Sami Hiltunen <shiltunen@gitlab.com>
Approved-by: James Fargher <jfargher@gitlab.com>
Co-authored-by: GitLab Renovate Bot <gitlab-bot@gitlab.com>

go: Update module github.com/rubenv/sql-migrate to v1.5.1

See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6002



Merged-by: Sami Hiltunen <shiltunen@gitlab.com>
Approved-by: Sami Hiltunen <shiltunen@gitlab.com>
Co-authored-by: GitLab Renovate Bot <gitlab-bot@gitlab.com>

In eb9491a0 (ci: Allow `test:macos` job to fail, 2023-04-28), we have
adapted our macOS job to fail. This was because we were frequently
observing flakes and errors due to various reasons.

This has since been fixed by migrating to GitLab's new macOS runners and
speeding up their tests significantly by using a RAM disk. Since then,
the job is significantly more stable and finishes faster than e.g. the
FIPS jobs.

Let's require the macOS job to succeed again. While not required for
production systems, it is important for many of GitLab's developers as
the majority does indeed use macOS on their development machines.

The NewBlobObject message was used as part of the ListNewBlobs RPC. The
RPC was removed in a305a1ea (ref: Drop ListNewBlobs RPC, 2021-09-21),
but missed to remove the NewBlobObject message. It is thus unused now.

Remove the message.

The GetBlob RPC does not carry any documentation for its Protobuf
definitions. Add them so that it becomes easier for callers to figure
out how the RPC is supposed to work.

The GetBlobs RPC does not carry any documentation for its Protobuf
definitions. Add them so that it becomes easier for callers to figure
out how the RPC is supposed to work.

The behaviour of GetBlobs is not exactly clear in all edge cases, which
makes it hard to document its Protobuf definitions correctly. Expand the
testcases to help with this.

Especially in our tests we often need to generate some arbitrary test
object IDs. This is needlessly complex right now and either requires
some fiddling with `git.ObjectHash.EncodedLen()` or using the hasher
interface to compute the object ID.

Provide a new `HashData()` function that hashes data to either a SHA1 or
SHA256 object ID.

As more and more tests of ours are gaining support for SHA256 it becomes
harder to figure out object sizes of some objects. While the object size
for a blob would be the same, all the other objects encode object IDs
and thus have different sizes for SHA1 and SHA256.

Add a new helper function `gittest.ObjectSize()` that retrieves the
object size via git-cat-file(1)'s `-s` option.

repository: Support SHA256 repositories for `GetSnapshot()`

See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6069



Merged-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: Justin Tobler <jtobler@gitlab.com>
Co-authored-by: Justin Tobler <jtobler@gitlab.com>

.gitlab-ci.yml: Test code only

See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6082



Merged-by: John Cai <jcai@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Reviewed-by: Justin Tobler <jtobler@gitlab.com>

Refactor small set of RPC tests to support SHA256

See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6052



Merged-by: James Fargher <jfargher@gitlab.com>
Approved-by: James Fargher <jfargher@gitlab.com>
Approved-by: karthik nayak <knayak@gitlab.com>
Reviewed-by: karthik nayak <knayak@gitlab.com>
Co-authored-by: Patrick Steinhardt <psteinhardt@gitlab.com>

Add -server-side option to gitaly-backup restore subcommand

See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6085



Merged-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: Evan Read <eread@gitlab.com>
Approved-by: karthik nayak <knayak@gitlab.com>
Reviewed-by: Evan Read <eread@gitlab.com>
Reviewed-by: karthik nayak <knayak@gitlab.com>
Co-authored-by: James Fargher <jfargher@gitlab.com>

Migrate objects to the repository's configured object directory

See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6048



Merged-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Reviewed-by: Sami Hiltunen <shiltunen@gitlab.com>
Reviewed-by: Justin Tobler <jtobler@gitlab.com>
Co-authored-by: Sami Hiltunen <shiltunen@gitlab.com>

Add only the configured object directory as an alternate

See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6060



Merged-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Reviewed-by: Sami Hiltunen <shiltunen@gitlab.com>
Reviewed-by: Justin Tobler <jtobler@gitlab.com>
Co-authored-by: Sami Hiltunen <shiltunen@gitlab.com>

walk: Move repo walking to 'walk' package

See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6067



Merged-by: karthik nayak <knayak@gitlab.com>
Approved-by: John Cai <jcai@gitlab.com>
Approved-by: karthik nayak <knayak@gitlab.com>
Reviewed-by: karthik nayak <knayak@gitlab.com>
Co-authored-by: Will Chandler <wchandler@gitlab.com>

Refactor the FilterShasWithSignatures tests to generate test data at
runtime so that we can test with the SHA256 object format.

Refactor the CommitLanguages tests to generate test data at runtime so
that we can test with the SHA256 object format.