[ci skip]

README: Add link to backpressure video

See merge request gitlab-org/gitaly!4475

Add a link in the presentations section of the README to a video
explaining the different ways to configure request limits (or
backpressure) in Gitaly.

Expose command stats (rusage) metrics via prometheus

See merge request gitlab-org/gitaly!4464

Update activesupport to 6.1.4.7

See merge request gitlab-org/gitaly!4471

For linguist, we prepend `env` as well as `bundle exec`
to the exec invocation.

For better accounting, we can override this, so that
it shows `git-linguist` instead of `env` as the cmd".

Handle DeleteObjectPool calls in Praefect

Closes #3742 and #4078

See merge request gitlab-org/gitaly!4395

Remove implicit pool creation on link behavior

See merge request gitlab-org/gitaly!4455

featureflag: Remove TransactionalSymbolicRefUpdates featureflag

See merge request gitlab-org/gitaly!4467

Add RateLimiting

Closes #4026

See merge request gitlab-org/gitaly!4427

Add a rate limiting middleware into the interceptor chain for a Gitaly
server.

Changelog: added

Now that we are adding a second limit handle, adjust the code to allow
for multiple limit handlers to be passed into a server invocation.

RateLimiter contains a limiter per rpc/repo pair. We don't want this to
grow monotinically since it will incur a heavy memory burden on the
machine. Instead, introduce a background process that looks through the
limiters and removes the ones that have not been used in the past 10
refill intervals.

Introduce a simple rate limiter that limits the number of requests a
minute that an RPC can allow.

If the feature flag is enabled, the middleware will drop any request
that bursts the per second limit of the RPC. Otherwise, it will only emit
metrics so we can first have some data on the traffic profile.

[ci skip]

This feature flag has been set to default on, and deleted from
production. There have been no observable issues, so it's now safe to
fully remove the feature flag.

Changelog: changed

commit: Add CheckObjectsExist RPC

Closes #3986

See merge request gitlab-org/gitaly!4450

When pushing commits to a repository, access checks are run. In order to
use the quarantine directory, we need a way to filter out revisions that
a repository already has in the case that a packfile sends over objects
that already exists on the server. In this case, we don't need to check
the access.

Add an RPC that when given a list of revisions, returns the ones that
already exist in the repository, and the ones that do not exist in the
repository.

Changelog: added

Allow Commit.RawBlame to take a Range parameter

See merge request gitlab-org/gitaly!4433

Before, concurrency limiting was the only limiting middleware. The name
of the metric had gitaly_rate_limiting in it, which was a bit of a
misnomer since rate was never part of the equation. Now that we are
actually adding a rate limiter to Gitaly, the concurrency metrics will
be mistaken for rate limiting metrics.

Change the name of these by replacing rate_limiting with
concurrency_limiting.

Changelog: changed

A future commit will add a new middleware that will limit based on the
rate rather than concurrent calls. There is a good amount of logic
currently used by the concurrency limiter that can be reused since a
rate limiter is also operating on incoming requests based on RPC name.

To make easier to add this new limiter type in the future, refactor the
code by adding some abstractions easier to add another type of limiter.

To prepare for a rate limiting middleware, add a struct to support
configuring a rate limiter. Behind the scenes, we are using
golang.org/x/time/rate package, which implements rate limiting with a
concept of a token bucket. There are two relevant values. Burst refers
to the maximum size of the token bucket. For a request to be handled, a
token is retrieved from the token bucket. Once the bucket is empty, no
more requests can be handled.

The token bucket will be refilled to capacity as defined by "Burst"
according to what is set by "Interval".

Changelog: added

We currently track rusage metrics in logs on a per-RPC basis. This
allows us to get a very fine-grained view into resource attribution.

However, logs often do not lend themselves to corse-grained and long-
term analysis. For this reason it is useful to expose metrics via
prometheus.

By aggregating that data as metrics, we aim to partially close an
observability gap that exists for short-lived processes. The existing
`process-exporter` metrics are severely under-reporting the utilization
of short-lived processes, which gitaly spawns many of.

See also:

- https://gitlab.com/gitlab-com/gl-infra/scalability#1655

This patch introduces a set of `gitaly_command_*` metrics which
provide aggregated resource attribution along the following
dimensions:

- `cmd` - the basename of the command being executed.
- `subcmd` - an optional subcommand, e.g. `archive` for `git archive`
- `grpc_service` - the grpc service caller
- `grpc_method` - the grpc method caller

The newly introduced metrics are:

- `gitaly_command_cpu_seconds_total` Sum of CPU time spent
- `gitaly_command_real_seconds_total` Sum of real time spent
- `gitaly_command_minor_page_faults_total` Sum of minor page faults
- `gitaly_command_major_page_faults_total` Sum of major page faults
- `gitaly_command_signals_received_total` Sum of signals received
- `gitaly_command_context_switches_total` Sum of context switches

This feature is being introduced behind a feature flag. However,
since metrics are sticky, once the metric has been defined, it
will be returned by the process until the next restart.

The cardinality of the metrics should be relatively well-bounded
in any case.

Git supports a wide array of range options for this option, but for now
we restrict the formats we support to the very simplest, which is of
the form "1,100".

We can use this to implement limits on how many lines of the blame we
retrieve GitLab-side, which should help to reduce the resource usage
of this particular RPC.

Changelog: added

Use rev-list --all --objects --disk-usage to calculate repository usage

See merge request gitlab-org/gitaly!4430

In the previous commit, we added a Size() method in localrepo that calls
git rev-list --all --objects --disk-usage to calculate the size of the
repository.

Add a feature flag that, when toggled on, will cause RepositorySize to
use this new way of calculating repository size.

Changelog: added

Add a Size() method for repo that calculates disk usage of objects. The
way we currently calculate repository size is to use du(1), which is a
quick and dirty way to get the total storage of a repository. However,
this ends up varying depending on the __how__ the objects are stored by
Git, which is affected by repository maintenance strategies and can vary.

A more consistent way to calculate storage is to get a total of number
of reachable objects with git rev-list --all --objects --disk-usage

ssh: Clean up output when pre-receive hook fails

See merge request gitlab-org/gitaly!4318

When the pre-receive hook failed, git-receive-pack(1) exits with code 0.
It's arguable whether this is the expected behavior, but anyhow it means
cmd.Wait() did not error out. On the other hand, the gitaly-hooks
command did stop the transaction upon failure. So this final vote fails.

To avoid this error being presented to the end user, ignore it when the
transaction was stopped.

Issue: https://gitlab.com/gitlab-org/gitaly/-/issues/3636
Changelog: fixed

Add test to check if the error message from a failing custom pre-receive
hook is presented to the user.

In a future commit we'd like to process the stdout and stderr output
from the ssh push command separately. With this commit the setting up of
the command is extracted into a function so in the future it's possible
to specify where to write stdout/stderr to.

Praefect is currently disabled in RemoveRepository tests which allows
for Praefect's RemoveRepository behavior to deviate from the Gitaly's.
This commit enables Praefect for these tests so we can be sure we catch
behavior deviations between the two handlers.

Praefect contains two special handlers for handling repository
removals. Both of these handlers do largely the same thing. This
commit shares the common code between the two handlers.

Praefect currently proxies DeleteObjectPool calls to Gitalys like
any other mutating RPC. This has the same problem as RemoveRepository
previously had; a pool can be deleted from the disk and it's metadata
record still left in the database. This can then cause problems as
Praefect believes a pool replica still exists where one doesn't exist.

Praefect doesn't even treat DeleteObjectPool as a repository removing
RPC. This means the deletions have never been replicated to the
secondaries and the pool metadata records have been left in place. This
then causes the reconciler to repeatedly attempt to reconcile from a
non-existing primary pool replica to the secondaries.

This commit fixes both issues by handling pool deletions in Praefect.
Similarly to RemoveRepository, the metadata record of the pool is first
deleted and only then the pool is attempted to remove from the Gitalys
that have it. This ensures atomicity of the removals when Praefect is
rewriting the replica paths.

With the behavior fixed, the Praefect specific test asserations can be
removed as the behavior now matches what how a plain Gitaly would
behave.

The handler in Praefect is tested via the same tests that test the
handler in Gitaly. Adding separate tests doesn't make sense as external
behavior of the the handlers should match and the handler shouldn't
exists in Praefect if it is removed from Gitaly.

Changelog: fixed

DeleteObjectPool's tests do not currently test what happens if the
request is missing the pool repository. This commit adds the missing
test case. While at it, it exports the functionality to extract a pool
from the request so Praefect's DeleteObjectPool handler can reuse it
later and pass the same test.

TestDelete does not verify whether the object pool exists or not
after the call to DeleteObjectPool. This is somewhat of an imporant
assertion to make to ensure the deletion logic is working correctly.
This commit adds that missing assertion.