mirror of
https://github.com/healthchecks/healthchecks.git
synced 2025-02-22 03:58:36 +00:00
b5d4f2aa74
The mitigation is to not attempt GetObject calls if there have been more than 3 S3 errors in the past minute. The implementation uses the TokenBucket class that we normally use for rate-limiting. An example scenario this is trying to avoid is: * the S3 service becomes unavailable for 10 straight minutes. Each S3 request hangs until we hit the configured timeout (settings.S3_TIMEOUT) * A client is frequently requesting the "Get ping's logged body" API call. Each call causes one webserver process to become busy for S3_TIMEOUT seconds. * All workers become busy, request backlog fills up, our service starts returning 5xx errors. With the mitigation, during an S3 outage, only the calls that retrieve ping's logged body will return 503, the rest of the service will (hopefully) work normally. Fixes: #1114 |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| test_badges.py | ||
| test_curl.py | ||
| test_date.py | ||
| test_emails.py | ||
| test_html.py | ||
| test_s3.py | ||
| test_signing.py | ||
| test_statsd.py | ||
| test_string.py | ||