limiter: Make CpuThrottledThreshold configureable
In the current implementation, the adaptive limiting kicks in when the resource level exceeds some hard-coded thresholds: * 90% of the parent cgroup's memory. * Cgroup's cpu is throttled for 50% of the observation time. In a recent incident, the limiter worked but it was triggered a bit late. When the memory level reaches 90%, the memory headroom might be tight. The inflight operations (usually expensive) can fill up the rest very quickly. When the memory level reaches 100%, a lot of weird things might occur, such as high memory pressure leading to major page faults, failed memory allocations, high iowait (because of page faults), OOM killing, etc. There's a chance that inflight requests cannot finish at this stage. So, it makes sense to increase this headroom by decreasing the threshold. This commit adds the ability to customize the CPU throttled threshold.
Showing
- internal/cli/gitaly/serve.go 1 addition, 1 deletioninternal/cli/gitaly/serve.go
- internal/gitaly/config/config.go 18 additions, 0 deletionsinternal/gitaly/config/config.go
- internal/gitaly/config/config_test.go 20 additions, 0 deletionsinternal/gitaly/config/config_test.go
- internal/limiter/watchers/cgroup_cpu_watcher.go 14 additions, 9 deletionsinternal/limiter/watchers/cgroup_cpu_watcher.go
- internal/limiter/watchers/cgroup_cpu_watcher_test.go 46 additions, 2 deletionsinternal/limiter/watchers/cgroup_cpu_watcher_test.go
Please register or sign in to comment