sts-sponsors team mailing list archive

[Alexsander de Souza <mp+443916@xxxxxxxxxxxxxxxxxx>]

Alexsander de Souza has proposed merging ~alexsander-souza/maas:lp1996204_fix_prometheus_labels into maas:master.

Commit message:
normalize request path used as prometheus label

using the original requested path effectively tracks metrics per database object, not per endpoint as intended. Additionally it leads to huge number of histograms in the prometheus database, exceeding the 2GB limit after some time.

fixes LP#1996204

Requested reviews:
  MAAS Maintainers (maas-maintainers)
Related bugs:
  Bug #1996204 in MAAS: "failing metrics cause 500 error"
  https://bugs.launchpad.net/maas/+bug/1996204

For more details, see:
https://code.launchpad.net/~alexsander-souza/maas/+git/maas/+merge/443916
-- 
Your team MAAS Committers is subscribed to branch maas:master.

diff --git a/src/maasserver/prometheus/middleware.py b/src/maasserver/prometheus/middleware.py
index 02b4867..607ad7d 100644
--- a/src/maasserver/prometheus/middleware.py
+++ b/src/maasserver/prometheus/middleware.py
@@ -6,6 +6,7 @@ from time import time
 
 from django.db import connections
 from django.db.backends.utils import CursorWrapper
+from django.urls import resolve, reverse
 
 from provisioningserver.prometheus.metrics import PROMETHEUS_METRICS
 
@@ -56,13 +57,21 @@ class PrometheusRequestMetricsMiddleware:
         return response
 
     def _process_metrics(self, request, response, latency, query_latencies):
-        op = request.POST.get("op", request.GET.get("op", ""))
         labels = {
             "method": request.method,
-            "path": request.path,
             "status": response.status_code,
-            "op": op,
+            "op": "",
+            "path": "",
         }
+        try:
+            match = resolve(request.path.removeprefix("/MAAS"))
+            args = [f":arg{i}" for i in range(len(match.args))]
+            kwargs = {k: f":{k}" for k in match.kwargs.keys()}
+            labels["path"] = reverse(match.url_name, None, args, kwargs)
+            labels["op"] = request.POST.get("op", request.GET.get("op", ""))
+        except Exception:
+            labels["path"] = request.path
+
         self.prometheus_metrics.update(
             "maas_http_request_latency",
             "observe",