This project is mirrored from https://gitlab.com/gitlab-org/gitaly.git.
Pull mirroring updated .
- Feb 21, 2022
-
-
Patrick Steinhardt authored
While we set up the housekeeping manager when starting Gitaly, we don't inject it into the service dependencies. As a result, Gitaly will not have it set up during normal runtime operations and will thus trigger a panic whenever trying to access it. This went unnoticed because our test setup for the Gitaly server is different from the setup we use when running tests. Fix this bug by injecting the housekeeping manager as expected Changelog: fixed
-
Patrick Steinhardt authored
housekeeping: Create central stateful architecture which handles repository optimizations See merge request gitlab-org/gitaly!4348
-
- Feb 18, 2022
-
-
John Cai authored
log: Disable gRPC HealthCheck message logging Closes #3428 See merge request gitlab-org/gitaly!4363
-
Sami Hiltunen authored
Create support request issue template See merge request gitlab-org/gitaly!3836
-
Stan Hu authored
gRPC HealthCheck messages are quite noisy, dominate the log volume in Gitaly, and usually are not that useful. We now disable them by default and add documentation on how to enable them. Closes https://gitlab.com/gitlab-org/gitaly/-/issues/3428 Changelog: changed
-
Sami Hiltunen authored
-
- Feb 17, 2022
-
-
Sami Hiltunen authored
datastore: Clean completed & dead replication jobs Closes #3665 See merge request gitlab-org/gitaly!4353
-
Patrick Steinhardt authored
It is currently hard to observe which optimizations were executed for repositories and how long they typically take. This commit introduces two new metrics to better track this information, which also allows us to easily write tests for OptimizeRepository.
-
John Cai authored
git: Backport patches to speed up git-fetch(1) in repos with many refs Closes #4050 See merge request gitlab-org/gitaly!4355
-
- Feb 16, 2022
-
-
Patrick Steinhardt authored
We have upstreamed two patches for git-fetch(1) which speed up mirror fetches in repositories with hundreds of millions of references. The first patch makes better use of the commit-graph when computing common objects, and the second patch will cause us to skip computing the output width when fetching with the `--quiet` flag. The patches were tested with our notorious problem repo `www-gitlab-com` with speedups of about 25% when doing mirror-fetches. Backport them into our own Git version now that they have been merged via 2b331293fb (Merge branch 'ps/fetch-optim-with-commit-graph' into next, 2022-02-14). Changelog: performance
-
Patrick Steinhardt authored
Now that we have the housekeeping manager globally injected as required, move the logic to clean up repositories and optimize them to this manager.
-
Patrick Steinhardt authored
We're eventually going to move the housekeeping package to keep state such that we can improve the heuristics used for optimizing repos. This will allow us to e.g. take into account the last time a repository was optimized to not optimize repositories too often, or avoid packing the same repository twice concurrently. Implment a new housekeeping manager structure that will be the home of this functionality and inject it as required. This manager isn't yet doing anything.
-
Patrick Steinhardt authored
We want to allow for OptimizeRepository to be called without invoking an RPC in order to allow us to call it in more contexts. Now that all its dependent parts are part of the housekeeping package this commit moves the logic of OptimizeRepository in there, too.
-
Patrick Steinhardt authored
We want to allow for OptimizeRepository to be called without invoking an RPC in order to allow us to call it in more contexts. As a preparatory step, we thus have to move all functionality which is invoked by it into a package that is independent of gRPC services. Move the logic handling repacking of objects into the housekeeping package as part of this move.
-
Patrick Steinhardt authored
We want to allow for OptimizeRepository to be called without invoking an RPC in order to allow us to call it in more contexts. As a preparatory step, we thus have to move all functionality which is invoked by it into a package that is independent of gRPC services. Move the writing of the commit-graph into the housekeeping package as part of this move.
-
Patrick Steinhardt authored
We want to allow for OptimizeRepository to be called without invoking an RPC in order to allow us to call it in more contexts. As a preparatory step, we thus have to move all functionality which is invoked by it into a package that is independent of gRPC services. Move the cleanup of worktrees into the housekeeping package as part of this move.
-
Patrick Steinhardt authored
We're about to move handling of all repository implementations into the housekeeping package. Prepare for this by making OptimizeRepository not depend on the RepositoryService server anymore.
-
- Feb 15, 2022
-
-
James Fargher authored
coordinator: Fix error comparison causing excessive replication jobs Closes #4045 See merge request gitlab-org/gitaly!4349
-
Pavlo Strokov authored
Propagate NotFound error returned by GetRepoPath See merge request gitlab-org/gitaly!4338
-
Toon Claes authored
Previous attempts [1] & [2] were made to avoid completed and dead jobs being left over in the replication queue. But only in [3] there was made sure no new jobs were generated. So now no more of these stale are created, we can run a migration that wipes all stale records from the database. This change in fact repeats the `DELETE` query from the migration added in [2]. 1. 8f8ae302 (Replication job acknowledge removes 'completed' and 'dead' events., 2020-08-05) 2. 2d3fc806 (Replication job acknowledge removes 'completed' and 'dead' events., 2020-09-03) 3. 8c4da135 (replication: Remove 'dead' stale jobs., 2021-10-20) Changelog: performance Issue: https://gitlab.com/gitlab-org/gitaly/-/issues/3665
-
Igor Wiedler authored
We are currently wrapping some of the errors returned by GetRepoPath, which makes them lose the gRPC NotFound code. This leads to a gRPC error code of Unknown, which counts towards our error SLO, paging the on-call. This patch removes the wrapping, ensuring the error code is propagated to the client, and accounted for appropriately in logs and metrics. Changelog: fixed
-
GitLab Release Tools Bot authored
[ci skip]
-
Patrick Steinhardt authored
limithandler: fix flaky test TestLimitHandlerMetrics See merge request gitlab-org/gitaly!4350
-
- Feb 14, 2022
-
-
John Cai authored
In this test, we didn't wait for both requests to return. Thus the test was ending and the grpc server was being shut down before the request could return. Fix this by putting both requests onto the request channel and waiting for both requests to be received before exiting the test.
-
John Cai authored
Remove metadata creation hack from Praefect Closes #4019 See merge request gitlab-org/gitaly!4337
-
Patrick Steinhardt authored
When determining whether nodes need replication jobs or not we also take into account the error status of a node: if the node returned an error that is different from the error returned by the primary node we create a replication job. The underlying assumption is that if two nodes behave the same, they should also run into the same kind of error. And if they returned different errors, then they likely did different things and may have diverged. This comparison is flawed though: we typically handle gRPC-style errors in this context, and those cannot be directly compared with each other. As a result, even in the case where two nodes returned the same error message and code we label them as different and thus create replication jobs. Fix this bug by manually comparing error code and message in case we've got a gRPC error. Note that we do not do this for normal Go errors: it is unexpected in the first place to get anything but a gRPC error, so we treat these as "weird" state and err on the side of caution. Changelog: fixed
-
Patrick Steinhardt authored
The tests which determine whether `getUpdatedAndOutdatedNodes()` works correctly is using standard Go errors to verify whether comparison of errors works as expected. This is not the typical kind of errors we'd get there though, but instead we'd usually get gRPC errors which cannot be directly compared with each other. Add some testcases for gRPC-style errors and record their current behaviour.
-
Toon Claes authored
migrate: Print execution time of migrations See merge request gitlab-org/gitaly!4308
-
Toon Claes authored
Update actionpack and related Ruby gems See merge request gitlab-org/gitaly!4347
-
Sami Hiltunen authored
limithandler: Add metrics for queue limiting See merge request gitlab-org/gitaly!4335
-
Patrick Steinhardt authored
repository: Add new RPC to prune unreachable objects Closes #4041 See merge request gitlab-org/gitaly!4346
-
Stan Hu authored
This fixes [CVE-2022-23633](https://github.com/advisories/GHSA-wh98-p28r-vrc9), but this is likely not an issue with Gitaly since Gitaly doesn't serve HTTP requests with Rails. * Diff: https://github.com/rails/rails/compare/v6.1.4.4...v6.1.4.6 Relates to https://gitlab.com/gitlab-org/gitlab/-/issues/352659 Changelog: changed
-
Patrick Steinhardt authored
Mutating maintenance-style RPCs have special handling in the coordinator and replicator. Implement it for the new PruneUnreachableObjects RPC.
-
Patrick Steinhardt authored
When rewriting the repository's history with the BFG Repo-Cleaner, then we potentially accumulate lots and lots of unreachable objects in the repository's object database. By default, we'd clean up those objects after two weeks, which is a rather long time to sit on such a huge number of objects. To fix this usecase we have thus gained a `prune` parameter in our GarbageCollect RPC call: if set, then we prune unreachable objects if they haven't been accessed during the last 30 mintues. The problem with this though is that GarbageCollect does a lot more than only pruning objects: it may end up packing objects or objects, writing commit-graphs, write bitmaps or some other things. All of these are things we want to control ourselves though, but we instead let git-gc(1) dictate how the repository is packed. We're thus about to deprecate all RPCs which directly influence how a repository is packed in favor of OptimizeRepository: this is our "black box" RPC that, from the viewpoint of the caller, does something with the repository to make it great again. And this is by design: callers should not control the way Gitaly handles repository maintenance. This highlights the need though for a new RPC call which _only_ prunes objects which have become unreachable to disentangle it from repository maintenance tasks. This commit thus introduces PruneUnreachableObjects, a new RPC which does exactly that: any unreachable loose object that hasn't been touched in the last 30 minutes is going to be pruned. Note that to make this work correctly, the caller has to do two RPC calls: the first RPC call to OptimizeRepository is required to unpack unreachable loose objects, and 30 minutes later they may prune these objects with a second call to PruneUnreachableObjects. This is no different from right now, even though it's hidden away and (naturally) used incorrectly by Rails: GarbageCollect would need to be called twice, first to explode unreachable objects into loose objects and then second with `prune=true` to prune them after half an hour. This is because Git will only ever consider loose objects for pruning, and the grace period is determined by inspecting its access time. So the way Rails does this is broken, and the new RPC call doesn't change that fact. This is a separate story though and nothing we can fix in Gitaly: we must retain the grace period to avoid repository corruption. Changelog: added
-
- Feb 11, 2022
-
-
Patrick Steinhardt authored
The OptimizeRepository RPC is missing in our protoregistry tests. Add it.
-
Patrick Steinhardt authored
The protoregistry tests have a map of RPCs and their expected type. This map isn't sorted though, which makes it hard to spot missing RPCs or find a proper place for new ones. Let's sort those alphabetically.
-
Toon Claes authored
ci: Upgrade CI images to Debian bullseye See merge request gitlab-org/gitaly!4340
-
Sami Hiltunen authored
Set and add error fields for git2go protocol which were missed See merge request gitlab-org/gitaly!4342
-
Stan Hu authored
Debian bullseye replaces buster as the latest stable version. We have upgraded Cloud Native GitLab to bullseye (https://gitlab.com/gitlab-org/build/CNG/-/merge_requests/888), so let's also upgrade CI to match. Since upgrading system libraries will likely break C extensions, we need to tie the cache key to the Debian version to ensure gems get recompiled for this platform. In addition, previously the PostgreSQL client used was whatever system default. In Debian buster, this was PostgreSQL v11. In Debian bullseye, the default is now PostgreSQL 13. For now, we keep version 11 to avoid making changes in the Praefect `structure.sql` file. Changelog: changed
-
James Fargher authored
Adding this field was missed in the original change 2af0319c
-