Signed-off-by: Rémy Coutable <remy@rymai.me>

Rewrite relative path in RepositoryReplicas

See merge request gitlab-org/gitaly!4384

git: Add support for bundled Git v2.35.1.gl0

Closes #4067 and #4068

See merge request gitlab-org/gitaly!4352

RepositoryReplicas RPC is currently not accounting for relative path
rewriting when sending the CalculateChecksum RPCs. This causes it to
fail when Praefect generates unique paths for the repositories. This
commit fixes the issue by sending the requests to the Gitalys with
the appropriate replica paths.

Praefect's test for RepositoryReplicas is not creating the repository
through the API. As such, it doesn't exercise the relative path
rewriting code and doesn't catch problems there. The QA tests caught
the fact that RepositoryReplicas is not doing relative path rewriting
yet. This commit updates the test first to create the repository through
the API and to account for the relative path rewriting when creating
a test commit in one of the replicas.

[ci skip]

Mark RefTransactionService as handled by Praefect

See merge request gitlab-org/gitaly!4383

This commit marks the RefTransactionService with the intercepted
option to denote it is handled by Praefect. Such services do not
need operation type annotations on the RPCs as they do not go through
the proxy. This should have no behavioral changes and mostly just
cleans up the code a bit.

We now have support in place for different bundled Git execution
environments, and a preceding commit has already started to build and
install bundled Git v2.35.1.gl1.

Add support for this new bundled Git execution environment behind a
feature flag.

Changelog: added

We're currently still stuck with Git v2.33.1.gl2 because we haven't yet
been able to roll out bundled Git to production, and we do want to make
use of feature flags to switch to newly introduced Git versions. Bundled
Git should soon be landing in production though, so this commit starts
to build and install Git v2.35.1.gl1 such that we're prepared to quickly
roll out bundled Git and then the new Git version.

Note that this also backports two patch series from the upcoming Git
v2.36.0 release:

    - Patches 21-26 fix an issue with the files refs backend, which
      would execute the reference-transaction hook for references that
      were about to be deleted twice: once for the loose ref itself, and
      once for the packed backend. This is a performance issue, but also
      required us to have a workaround that filters out any reference
      transactions which only consisted of deletes.

    - Patches 27-34 fix the `--atomic` flag of git-fetch(1) to also
      cover deletion of references and backfilling of tags. This has
      caused excessive voting via reference transactions in production
      because the reference-transaction hook was executed twice for each
      reference modified via those mechanisms. It is thus a correctness
      fix because we don't commit changes to disk anymore if any of the
      other ref updates fails, and a performance fix for mirror fetches
      because we perform less voting.

We are about to introduce the ability to build multiple different Git
versions, and this will require us to also have multiple different
source directories. This commit prepares for this by converting our
current targets which build Git into pattern-based targets such that it
is easily possible to use multiple different directories.

Note that this commit also changes the default source directory of the
current default "_build/deps/git" one to "_build/deps/git-v2.33.1.gl2".
The build directory shouldn't be accessed by any callers, so this is not
expected to be much of a problem. What is accessed though is the default
installation directory for Git, which is "_build/deps/git/install". This
location is thus retained to not break compatibility.

The downside is that we'll now have to download multiple copies of the
Git repository. On the other hand, we now don't have to rebuild Git
whenever we change between different branches which use different Git
versions because the source directories would ideally not be modified
anymore.

We're about to remove references to `GIT_SOURCE_DIR` because there will
soon be multiple different source directories, where we have one per Git
version we're building. Prepare for this by converting recipes to derive
the location via automatic Makefile variables.

We have split up computation of the Git version and its patches into two
different sites, even though we only ever use patches in case the user
didn't pass a Git version or explicitly asked us for the "default" Git
version. This needlessly complicates the setup by requiring another
variable which tells us whether we have to apply patches or not.

Merge both sites into one to simplify the code.

When building Git we define both the GIT_PATCHES and GIT_EXTRA_VERSION
variables to influence the way Git is built. This is currently done via
`ifdef` conditionals, which are notably evaluated at the time of parsing
the Makefile. As a consequence, even if their values change at a later
point, we'll already have stripped or retained the code snippets they're
guarding because they won't be reevaluated. This is not an issue
currently, but it is going to be an issue when we ruse the target to
build multiple different Git versions.

Prepare for this by dynamically testing whether those variables are set
or not as part of the recipe itself.

ruby: Drop more unused code

See merge request gitlab-org/gitaly!4380

Add skip_content param to wiki find page call

See merge request gitlab-org/gitaly!4373

localrepo: Unlock symbolic refs when the update fails

Closes #4075

See merge request gitlab-org/gitaly!4379

[ci skip]

git: Prepare for multiple bundled Git versions

See merge request gitlab-org/gitaly!4372

[ci skip]

proto: Add structured error types for UserSquash RPC

See merge request gitlab-org/gitaly!4381

We're about to introduce support for structured errors in the UserSquash
RPC. We cannot roll out this change even with a feature flag though
because some of our testing infrastructure will always test code with
all feature flags enabled. We thus have to land the protobuf changes
independently such that we can adjust downstream callers.

This commit introduces these changes by creating a new UserSquashError
Protobuf message which contains all potential structured errors we want
to return from the UserSquash RPC.

Changelog: added

Add a short documentation snippet to UserSquash so that we can get rid
of the previous linting exception.

Originally when UserSquash was still using worktrees to do its work, we
used a squash ID which enabled us to reuse worktrees. We have since
converted the RPC to do the rebase in-memory though, which allowed us to
get rid of the use of worktrees and thus make the squash ID obsolete. It
was thus deprecated in 112d7472 (operations: Deprecate `SquashId`,
2021-08-20), and callsites in Rails have been converted in fc5b05874f9
(gitaly_client: Drop use of squash ID, 2021-09-21). There aren't any
other callsites left.

Remove the `squash_id` altogether.

Changelog: removed

This commit drops more code that is not being used anymore in the Gitaly
side car.

We use a `safe.LockingFileWriter` to get transactional semantics when
updating symbolic references. This mechanism will create a lock for the
file that is about to be updated, votes on the contents of the file and
then finally unlocks it if everything went alright. In case something
goes wrong though the caller must make sure to close the writer, or
otherwise we may leak the lockfile and thus also obstruct all subsequent
writes to that symbolic reference. We don't do this correctly though.

Fix this bug by always unlocking the locking file writer.

Changelog: fixed

We have recently added transactional support for writing symbolic
references in a repository via the `safe.LockingFileWriter` interface.
We forgot to unlock the writer in all cases though, which means that in
some error conditions we may leave behind a lockfile.

Add a testcase which demonstrates this failure to unlock.

We do not verify that all expected bundled Git binaries exist and are
executable when they have been configured by the administrator. As a
result, Gitaly will happly start up but then later fail at runtime when
trying to use the bundled Git execution environment in case any of the
binaries is missing.

Tighten our sanity checks to verify that the bundled Git environment is
set up correctly.

Changelog: fixed

When bundled Git is configured then the binaries are expected to be
contained in Gitaly's binary directory. It is thus mandatory for the
directory to be set, or else the resulting execution environment cannot
actually be a valid one. And while we already reject the case where the
binary directory is unset in case the `GITALY_TESTING_BUNDLED_GIT_PATH`
environment variable is set, we don't have such a test for "normal"
setups.

Add the missing sanify check.

Changelog: fixed

We're about to introduce another bundled Git version. Prepare tests
which exercise the execution environments to cope with this change.

While we only have a single bundled Git environment right now, the plan
is to extend support such that we can have multiple bundled environments
at the same time which are then feature-flaggable. The different sets of
binaries can be distinguished by their version suffix: while the current
set of binaries is simply called "gitaly-git" et al, we already have a
second version which installs as "gitaly-git-v2.35.1.gl1".

Allow bundled Git execution environments to specify version suffixes
such that it's easily possible to set up multiple ones.

While we only have a single bundled Git environment right now, the plan
is to extend support such that we can have multiple bundled environments
at the same time which are then feature-flaggable. The different sets of
binaries can be distinguished by their version suffix: while the current
set of binaries is simply called "gitaly-git" et al, we already have a
second version which installs as "gitaly-git-v2.35.1.gl1".

Allow bundled Git execution environments to specify version suffixes
such that it's easily possible to set up multiple ones.

The knowledge about which execution environments exist is distributed
across multiple different locations: first we need to set them up, and
second we need to make use them depending on whether they have been
configured and enabled. This is really fragile and ultimately hard to
maintain.

We now have the logic to set up the different environments in self
contained structures which all have the same interface. It is thus
trivial to move this logic into a single place, which is a new array of
all supported execution environment constructors. The Git command
factory can thus be stripped of all knowledge around which different
execution environments exists and only needs to loop through all
constructors -- if at least one of them returns a valid environment,
then we're good to go.

We need to maintain two properties though:

    - We prioritize the order in which we consult the different
      environments. If bundled Git is configured, then it should be
      preferred over distributed Git. And if distributed Git is
      configured, it should be preferred over the fallback Git
      environment which was resolved via PATH.

    - Environments must be feature-flaggable. So even if an environment
      with a higher priority is available, it may still be disabled
      which would thus cause us to pick a lower-priority backend.

These properties are maintained by encoding the priority of backends via
the order in which environment constructors are inside their array. The
Git command factory thus constructs all environments in the order they
are listed in and will then loop through them. The first backend that is
enabled (that is, all its feature flags are turned on) will be picked.

Like this, it is trivial to add new execution environments as they are
all tracked in a single central location.

We return a cleanup function whenever we construct Git execution
environments. Move this functionality into the environments themselves
to have less state to keep track of. Furthermore, this prepares for a
refactoring where we rework the way Git execution environments are
constructed in the first place.

Move the logic which creates execution environments for fallback Git
into its own structure. This will eventually allow us to have a simple
list of all supported Git execution environments in a central location.

Move the logic which creates execution environments for bundled Git into
its own structure. This will eventually allow us to have a simple list
of all supported Git execution environments in a central location.

Move the logic which creates execution environments for distributed Git
into its own structure. This will eventually allow us to have a simple
list of all supported Git execution environments in a central location.

When Praefect receives a ServerInfo RPC it forwards the request to
proxied Gitaly nodes to obtain their information and then pass it
through back to the caller. We do not forward the RPC metadata correctly
though, and as a result any feature flags passed to Praefect will not
end up with Gitaly.

Fix this by converting the incoming to an outgoing context before doing
the RPC calls to Gitaly.

Changelog: fixed

localrepo: Remove flag to switch to sidechannels for internal fetches

Closes #4065

See merge request gitlab-org/gitaly!4375