Return ReferenceUpdateError from UserMergeBranch with transactions

Closes #5995

See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6842



Merged-by: Will Chandler <wchandler@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: Sami Hiltunen <shiltunen@gitlab.com>
Reviewed-by: James Liu <jliu@gitlab.com>
Co-authored-by: Sami Hiltunen <shiltunen@gitlab.com>

Only disable calls into Rails with GITALY_TESTING_NO_GIT_HOOKS

Closes #5979

See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6830



Merged-by: John Cai <jcai@gitlab.com>
Approved-by: Karthik Nayak <knayak@gitlab.com>
Approved-by: John Cai <jcai@gitlab.com>
Reviewed-by: Karthik Nayak <knayak@gitlab.com>
Co-authored-by: Sami Hiltunen <shiltunen@gitlab.com>

We've copied the full names of various methods name in the
transaction middleware. Use the generated constants instead so we'll
have compilation failures if the RPC names change or the RPCs are
removed.

FetchObjectPool was previously typoed in the non-transactional list.
Now we actually don't start a transaction for it. Tests were adjusted
as the error messages are now the ones that are raised by the handler.

UserMergeBranch is not returning an error detail when it fails due to
a concurrent reference update and transactions are enabled. The error
is different as the RPC handler has finished by the time we commit the
transaction and encounter the conflict. This commit adds handling for
this case in the transaction middleware so we conform to the API of
UserMergeBranch when committing the transaction fails due to a
concurrent reference update.

TransactionManager returns a ReferenceVerificationError when a
reference's old OID didn't match what was expected by the transaction
attempting to update the reference. In some contexts it would be
useful to have the new OID of the attempted update also available.
Include the new OID in the error so it can be bubbled up.

Add Sign field to commit UserCommitFiles RPC

See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6795



Merged-by: John Cai <jcai@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Approved-by: John Cai <jcai@gitlab.com>
Reviewed-by: Igor Drozdov <idrozdov@gitlab.com>
Reviewed-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: John Cai <jcai@gitlab.com>
Co-authored-by: Igor Drozdov <idrozdov@gitlab.com>

Don't error out when deleting a non-existent pool with transactions

Closes #5989

See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6839



Merged-by: Karthik Nayak <knayak@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Approved-by: Karthik Nayak <knayak@gitlab.com>
Co-authored-by: Sami Hiltunen <shiltunen@gitlab.com>

go: Update module github.com/pelletier/go-toml/v2 to v2.2.1

See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6840



Merged-by: James Liu <jliu@gitlab.com>
Approved-by: Christian Couder <chriscool@tuxfamily.org>
Approved-by: James Liu <jliu@gitlab.com>
Co-authored-by: GitLab Renovate Bot <gitlab-bot@gitlab.com>

When transactions are enabled, Gitaly returns a not found error when
attempting to delete a non-existent object pool. Without transactions,
Gitaly returns a successful response. This was missed as the test case
testing non-existent pool deletion is actually deleting a pool that
exists.

Fix the broken test and adjust behavior with transactions to return
the same successful response as without. Arguably returning a not
found would be a better interface this will fix Rails pipeline failures
with transactions for now.

housekeeping/manager: Add housekeeping jobs to remove info/attributes

Closes #4540

See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6781



Merged-by: Eric Ju <eju@gitlab.com>
Approved-by: Karthik Nayak <knayak@gitlab.com>
Reviewed-by: Ahmad Sherif <ahmad@gitlab.com>
Reviewed-by: Karthik Nayak <knayak@gitlab.com>
Reviewed-by: Eric Ju <eju@gitlab.com>

Makefile: bump Git version to v2.44.0.gl2

See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6832



Merged-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Approved-by: Christian Couder <chriscool@tuxfamily.org>
Approved-by: Eric Ju <eju@gitlab.com>

featureflag: Default enable `autocrlf_false`

See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6838



Merged-by: James Liu <jliu@gitlab.com>
Approved-by: James Liu <jliu@gitlab.com>
Co-authored-by: Justin Tobler <jtobler@gitlab.com>

Support non-UTF-8 file system paths in logging protocol

See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6835



Merged-by: James Liu <jliu@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: James Liu <jliu@gitlab.com>
Reviewed-by: Sami Hiltunen <shiltunen@gitlab.com>
Reviewed-by: Justin Tobler <jtobler@gitlab.com>
Co-authored-by: Sami Hiltunen <shiltunen@gitlab.com>

The `autocrlf_false` feature flag, when enabled, set the `core.autocrlf`
Git config to `false`. This prevents Git from mutating line-endings for
commits written within Gitaly.

Default enable this feature flag.

diff: Support rename detection in `FindChangedPaths` RPC

See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6827



Merged-by: Will Chandler <wchandler@gitlab.com>
Approved-by: Karthik Nayak <knayak@gitlab.com>
Co-authored-by: Justin Tobler <jtobler@gitlab.com>

This field controls whether a commit created by Gitaly must
be signed.

It will allow us to skip commit signing for the calls that
support Author field customization, for example, Commits API

Changelog: added

Linux allows for non-UTF-8 file system paths. Our logging protocol
is currently storing paths as strings which in protocol buffers are
expected to be UTF-8. To allow for non-UTF-8 paths, convert the
protocol buffer types to bytes and adjust the code around it.

Tests covering the UTF-8 support exclude macOS as macOS enforces
UTF-8 in paths.

HookManager is created earlier than needed. Move the creation closer
to where it is needed to make it clearer where it is needed.

GITALY_TESTING_NO_GIT_HOOKS is an environment variable that disables
hooks in Gitaly entirely. This is only used by Rails' specs to disable
the calls into internal API as it is not set up during tests. Gitaly
however disables the entire hook functionality. This leads to us
testing less but causes problems with transactions.

Transactions rely on the reference transaction hook to capture the
reference updates made in the transaction. As the reference transaction
hook is not ran, the changes are not captured and are not committed.
Fix this issue by only stubbing out the Rails calls when the environment
variable is set. This leads us to running the full hook logic but just
stubbing out requests into Rails and assuming they were succcessful.
This tests Gitaly closer to how it runs in production and also fixes the
issue with transactions as reference transaction hook is running again.

Since the hooks were disabled, Rails' specs also haven't been exercising
Praefect's voting logic which should now be exercised. Praefect has so
far functioned mostly by accident as there is only a single voter, and
the transaction has concluded when the primary returns without voting.

Linux supports file system paths with invalid UTF-8 where as macOS
does not. Git allows non-UTF-8 reference names which may be written
out as file names if the reference is stored as a loose reference.
We'll soon add support for this, so we need a way to exclude the tests
from running on macOS.

service: skip flaky tests

See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6831



Merged-by: Karthik Nayak <knayak@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>

go: Update module github.com/grpc-ecosystem/go-grpc-middleware/v2 to v2.1.0

See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6724



Merged-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Ahmad Sherif <ahmad@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Co-authored-by: GitLab Renovate Bot <gitlab-bot@gitlab.com>

Remove ApplyGitattributes from WAL exclusion list

See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6829



Merged-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Eric Ju <eju@gitlab.com>

We have recently tried to roll out Git v2.44.0 to production systems but
discovered that it breaks object reuse in repositories with a multi-pack
index. This issue led to user-visible errors as well as object reuse
being disabled completely when serving fetches.

We have fixed the issue upstream and backported the fix to our custom
Git v2.44.0.gl2. Upgrade to that version to unblock the rollout.

From git 2.43 on, Gitaly will read from HEAD:.gitattributes instead of
.info/attributes. Old operation that copies HEAD:.gitattributes into
info/attributes will be deprecated. The old info/attributes file must
be removed because it has higher precedence than HEAD:.gitattributes.

test: Modify UserApplyPatch to assert on substring

See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6828



Merged-by: Karthik Nayak <knayak@gitlab.com>
Approved-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Approved-by: Karthik Nayak <knayak@gitlab.com>
Reviewed-by: Karthik Nayak <knayak@gitlab.com>
Co-authored-by: James Liu <jliu@gitlab.com>

Handle accessing revision and path in empty repository in FindCommits

Closes #5976

See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6826



Merged-by: Sami Hiltunen <shiltunen@gitlab.com>
Approved-by: Karthik Nayak <knayak@gitlab.com>
Reviewed-by: James Liu <jliu@gitlab.com>

The tests:
- TestProcReceiveHook
- TestServer_PostUploadPackWithBundleURI/backup_without_bundle_path
are flaky and keep failing in our CIs. Let's mark them flaky and skip
them till we get around to fixing them.

diff: Introduce `DiffBlobs` RPC

See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6808



Merged-by: Toon Claes <toon@gitlab.com>
Approved-by: Karthik Nayak <knayak@gitlab.com>
Approved-by: Toon Claes <toon@gitlab.com>
Reviewed-by: Justin Tobler <jtobler@gitlab.com>
Reviewed-by: Karthik Nayak <knayak@gitlab.com>
Co-authored-by: Justin Tobler <jtobler@gitlab.com>

test: Support reftables in misc filesystem accesses

See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6704



Merged-by: Karthik Nayak <knayak@gitlab.com>
Approved-by: Emily Chui <echui@gitlab.com>
Approved-by: Karthik Nayak <knayak@gitlab.com>
Reviewed-by: Emily Chui <echui@gitlab.com>
Co-authored-by: James Liu <jliu@gitlab.com>

Fix error type to match `sendTreeEntries` interface

See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6773



Merged-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Sami Hiltunen <shiltunen@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: Sami Hiltunen <shiltunen@gitlab.com>
Co-authored-by: Timo Furrer <tfurrer@gitlab.com>

[ci skip]

FindCommits previously didn't return an error when accessing an
empty repository with a given revision and path. This was because the
RPC previously didn't return errors from the Git command to the client.
The error handling was fixed in 588d901f
which led to the errors now being returned to the client. This leads to
Rails API returning internal server errors to clients as previously
working API invocations now started returning internal errors. Fix this
by handling the error that is raised by Git in such a scenario and
returning a NotFound error to Rails that it knows to handle.

Changelog: fixed

FindCommits tests are missing test cases looking for a specific
revision or path in an empty and non-empty repository. Add the
missing tests as them missing led to a regression.