I have a simple API application running and I'm using a custom Post manager to 'PUT' here periodically. My concern is that some of the 'PUTS' are taking over 50ms to complete . In Grafana I setup the following prom QL query to try and find these requests
(sum(rate(http_server_requests_seconds_bucket{exception="None", app="Myapp", method="PUT"}[5m])) by (le) > 0.05)
It seems from the results i get back this is pulling in all buckets showing me all of them for the value of 0.05556? grafana-screenshot but not actually showing me all requests that are over 50ms.
1 Answer
The http_server_requests_seconds_bucket metrics count the number of requests since last service restart with durations smaller or equal to the le label value. For example, http_server_requests_seconds_bucket{le="0.1"} counts the number of requests with durations smaller or equal to 0.1 seconds (e.g. 100ms).
The number of buckets and their boundaries are set by the application, which exposes the histogram. Buckets' count and buckets' boundaries affect the precision of the histogram. For example, if the http_server_requests_seconds histogram exposes only the following le boundaries: 0.01, 0.1, 1 and Inf, then it is impossible to calculate with 100% precision the number of requests with the duration exceeding 50ms. But it is possible to calculate the exact number of requests with durations bigger than 10ms, 100ms or 1s.
Note that http_server_requests_seconds_bucket metrics expose per-bucket counters, which start counting requests since the service restart. If you need to obtain per-bucket request distribution over some time window, then you need to wrap the http_server_requests_seconds_bucket into increase function with the specified time window in square brackets. For example, the following query returns per-bucket request distribution over the last hour:
increase(http_server_requests_seconds_bucket[1h])P.s. see also .
2
