Skip to content
Snippets Groups Projects
This project is mirrored from https://gitlab.com/gitlab-org/gitaly.git. Pull mirroring updated .
  1. Apr 26, 2022
    • John Cai's avatar
      proto: Add DeleteRefErrors · 49497a00
      John Cai authored
      In the DeleteRefs RPC, there are some cases when we do not return an
      error but instead put the error inside of DeleteRefsResponse. This is
      problematic because we may end up not voting, which leads to unnecessary
      replication jobs being created.
      
      This commit adds the protobuf definition for a structured error that we
      will return.
      49497a00
  2. Apr 25, 2022
  3. Apr 22, 2022
  4. Apr 21, 2022
  5. Apr 20, 2022
    • Sami Hiltunen's avatar
      Release expired verification leases periodically · e3ec6e29
      Sami Hiltunen authored
      The background verifier sets a lease time on a replica when it picks
      it up for verification. If the worker dies for some reason, the lease
      will remain in place and no other worker will pick up the replica for
      verification again until the lease is cleared. The lease itself tells
      the maximum time the worker itself would be working on the replica.
      After it has been passed, it would be safe for another worker to pick
      up the replica for verification again. This commit adds a background
      goroutine that periodically releases expired leases so other workers
      can take up the work if the original worker failed and did not release
      the lease. The 'verificaton_leases' index is added so the query can
      efficiently find the replicas with leases acquired to find the stale
      ones.
      e3ec6e29
    • James Fargher's avatar
      Merge branch 'eread/update-version-of-danger-files-dependency' into 'master' · 53768dae
      James Fargher authored
      Update version of danger-files dependency
      
      See merge request gitlab-org/gitaly!4485
      53768dae
    • Evan Read's avatar
      Update version of danger-files dependency · 811eb4dd
      Evan Read authored
      811eb4dd
    • John Cai's avatar
      Merge branch 'jc-docs-backpressure' into 'master' · 0ef7c067
      John Cai authored
      docs: Document Gitaly backpressure
      
      See merge request gitlab-org/gitaly!4469
      0ef7c067
    • John Cai's avatar
      docs: Document Gitaly backpressure · 051d510a
      John Cai authored
      There are a number of knobs in Gitaly to tune backpressure Gitaly can impose on services that call it. This commit documents these.
      051d510a
  6. Apr 19, 2022
    • James Fargher's avatar
      Merge branch 'sh-fips-mode' into 'master' · 5591e2b5
      James Fargher authored
      Add support for FIPS encryption
      
      See merge request gitlab-org/gitaly!4482
      5591e2b5
    • John Cai's avatar
      Merge branch 'jc-list-repo-to-use-shorter-grace-period' into 'master' · d0809beb
      John Cai authored
      repocleaner: Allow NewWalker to receive grace period parameter
      
      Closes #4164
      
      See merge request gitlab-org/gitaly!4474
      d0809beb
    • John Cai's avatar
      praefect: Set 6 hour grace period in list_untracked_repositories · 6b2ab5c3
      John Cai authored
      A default grace period of 6 hours is sufficient for the subcommand.
      6b2ab5c3
    • Sami Hiltunen's avatar
      Merge branch 'smh-verify-subcmd' into 'master' · 752243b8
      Sami Hiltunen authored
      Implement 'praefect verify' subcommand
      
      Closes #4091
      
      See merge request gitlab-org/gitaly!4463
      752243b8
    • Sami Hiltunen's avatar
      Implement 'praefect verify' subcommand · b902b6bf
      Sami Hiltunen authored
      Praefect periodically verifies the repository metadata in the background.
      The interval may be too long especially if there was an incident, say
      a disk failure. After recovering the Gitaly node, the disk may be completely
      empty or may contains an older snapshot which does not contain all expected
      repositories. In such cases, it would be great if Praefect could be manually
      instructed to verify the storage again as soon as possible rather than waiting
      for the next scheduled verification interval to pass. This commit adds the
      'praefect verify' subcommand that allows for doing that. It takes in either a
      repository id, a virtual storage or a (virtual storage, storage) tuple and
      marks all replicas matching the selector as being unverified. Praefect's
      metadata verifier will then prioritize verifying these repositories over other
      repositories pending verification. This allows administrators to speed up
      the verification process and thus recovery.
      
      Changelog: added
      b902b6bf
    • Sami Hiltunen's avatar
      Add proto definitions for MarkUnverified RPC · b850c399
      Sami Hiltunen authored
      With the introduction of metadata verification, Praefect needs a tool
      to manually mark a repository as needing verification immediately rather
      than after the specified verification interval has passed. That tool will
      require a new RPC that it can call achieve its goal. This commit adds the
      proto definitions for MarkUnverified RPC which can be called to either
      mark a single repository by ID, a whole virtual storage, or a whole storage
      as needing verification.
      
      Changelog: added
      b850c399
    • Sami Hiltunen's avatar
      Merge branch 'jc-limit-error-proto' into 'master' · ef18681a
      Sami Hiltunen authored
      proto: Add LimitError as a structured error
      
      See merge request gitlab-org/gitaly!4476
      ef18681a
  7. Apr 17, 2022
  8. Apr 14, 2022
    • Toon Claes's avatar
      Merge branch 'toon-go-fixes' into 'master' · fa2d37aa
      Toon Claes authored
      Makefile: Make GITALY_EXECUTABLES deferred again
      
      See merge request gitlab-org/gitaly!4477
      fa2d37aa
    • Toon Claes's avatar
      Merge branch 'jc-update-gitaly-toml-with' into 'master' · 6ff441e3
      Toon Claes authored
      Add new examples for concurrency and rate limiters
      
      See merge request gitlab-org/gitaly!4472
      6ff441e3
    • Toon Claes's avatar
      Makefile: Make GITALY_EXECUTABLES deferred again · c543e036
      Toon Claes authored
      Recently, in b5c9c7ef (Makefile: Rename find_commands to
      GITALY_EXECUTABLES, 2022-03-25), we've changed the variable that holds
      the names of all Gitaly executable to be an immediate variable. While
      this is a good idea in general, it causes trouble in CI. In CI the
      compiled executables are put in cache, but the source files are not. So
      when files are pulled from cache, and any make target is built, it will
      expand GITALY_EXECUTABLES. Now source files are not pulled from the
      cache, so the `cmd` directory is missing. And therefore we revert it
      back to be a deferred variable.
      c543e036
    • John Cai's avatar
      proto: Add LimitError as a structured error · 4e2d5220
      John Cai authored
      When Gitaly enforces a limit, either due rate limiting or concurrency
      limiting, it needs to be able to return an error to its clients to
      provide context into why it failed so that clients can then inform its
      callers of why the call failed.
      
      Changelog: added
      4e2d5220
    • Sami Hiltunen's avatar
      Merge branch 'smh-verify-metadata-output' into 'master' · c311109b
      Sami Hiltunen authored
      Expose last verification time in 'praefect metadata'
      
      Closes #4092
      
      See merge request gitlab-org/gitaly!4466
      c311109b
  9. Apr 13, 2022
    • Sami Hiltunen's avatar
      Ignore verification columns for read-only cache updates · 5c9feb4e
      Sami Hiltunen authored
      Read-only cache receives invalidations on record updates via triggers
      in Postgres. Currently the notifications are sent for any modification
      to the records. The verification related columns are not relevant to
      the operation of the cache so this commit ignores the changes to the
      columns in the triggers.
      
      Changelog: changed
      5c9feb4e
    • Sami Hiltunen's avatar
      Expose last verification time in 'praefect metadata' · 4e9fc294
      Sami Hiltunen authored
      Administrator's may want to know when Praefect has last verified a
      replica. This commit exposes that information via the 'praefect metadata'
      command.
      
      Changelog: changed
      4e9fc294
    • Sami Hiltunen's avatar
      Fetch verified_at as part of GetRepositoryMetadata · 170c4828
      Sami Hiltunen authored
      GetRepositoryMetadata fetches a repository's metadata from the
      database. This commit expands the query to also fetch the newly added
      verified_at column so we can expose it in the 'praefect metadata'
      command to the admins.
      170c4828
    • Sami Hiltunen's avatar
      Expose VerifiedAt via GetRepositoryMetadata proto definitions · 107b54cd
      Sami Hiltunen authored
      Administrators may want to know when a replica has been last verified
      by Praefect. GetRepositoryMetadata RPC is called by the 'metadata'
      sub-command to retrieve infromation about a repository and its
      replicas from Praefect's database. This commit adds the proto
      definitions for exposing the last verification time of replicas to
      the metadata sub-command.
      
      Changelog: changed
      107b54cd
    • Sami Hiltunen's avatar
      Merge branch 'smh-background-verifier' into 'master' · 9413ca59
      Sami Hiltunen authored
      Initial implementation of a metadata verifier
      
      See merge request gitlab-org/gitaly!4459
      9413ca59
    • Sami Hiltunen's avatar
      Wire metadata verifier in Praefect's main · 85ace7cf
      Sami Hiltunen authored
      This commit wires the metadata verifier in Praefect's main so it can
      actually be configured for use. It's default disabled still as it still
      is missing some functionality that should be in place before generally
      enabling it, for example tooling like metrics, integration in to the
      'praefect metadata' tool and a background routine to release stale leases.
      
      Changelog: added
      85ace7cf
    • Sami Hiltunen's avatar
      Initial implementation of a metadata verifier · 6fce7a80
      Sami Hiltunen authored
      This commit adds an initial implementation of a metadata verifier
      to Praefect.
      
      Praefect stores metadata of the repositories stored on the cluster in
      Postgres. These metadata records may become out of sync with the disks
      if changes occur on the disks without going through Praefect, for example
      due to disk failures or manual modifications. Right now, Praefect only
      contains some temporary logic to clean up invalid metadata records when
      replication is attempted using a non-existent source repository. This was
      mostly put in place to stop reconciliation loops where Praefect keeps
      scheduling replication jobs from the non-existent repository that will
      never succeed. While this performs some clean up, it's not sufficient to
      catch cases where something happens in the background without prompting
      replication.
      
      The metadata verifier introduced in this commit aims to catch these issues
      by verifying the metadata eveynow and then in the background with the
      state on the disks. For now, only the existence of the replica is verified,
      not the actual contents by checksumming.
      
      Each replica contains a 'verified_at' timestamp in the database that tells
      Praefect when the metadata record was last verified. If it exceeds a configurable
      threshold, the replica is considered to be due for reverification. Praefect
      then asks the Gitaly hosting the replica whether the replica still exists.
      If it doesn't the invalid metadata record is deleted and the removal is logged.
      To avoid multiple Praefects verifying the same replica concurrently, Praefect
      acquires the verification lease on the replica in the database prior to
      verifying the existence of the repository.
      
      The scheduling is fairly simplistic at the moment with each Praefect acquiring
      a batch of work every two seconds. This also serves as a crude way to rate
      limit the background verification work rather to avoid consuming too many
      resources while doing it. This should be sufficient for now althoug could later
      be improved.
      
      Praefect leaves the repository's record in place even if all of its replicas
      have been lost. This ensures no data loss goes unnoticed and that the loss
      needs to be acknowledged by removing the repository manually.
      
      Changelog: added
      6fce7a80
    • Sami Hiltunen's avatar
      Add migrations for background verification schema · 465af671
      Sami Hiltunen authored
      This commit adds the necessary schema changes for the metadata
      background verification. Each replica receives two new columns:
      
      1. 'verified_at' which contains the timestamp of the last successful
         verification of the replica. This effectively allows for identifying
         replicas that are in need of reverification.
      
      2. 'verification_leased_until' which contains a timestamp until which
         a worker has acquired a lease to reverify the repository. This prevents
         multiple workers from picking the same repository for reverification at
         the same time.
      
      'verification_queue' index is added to index replicas which have not been
      acquired by any worker. This allows for efficientl querying replicas that
      are in need of reverification later.
      
      Changelog: other
      465af671