Zum Inhalt

Prometheus for Application Metrics

Einordnung ins Overtime-Projekt

Neben Logs braucht Overtime auch Echtzeit-Metriken – etwa zur Zahl aktiver WebSocket-Verbindungen oder gleichzeitiger Video-Calls –, um Kapazität und Verfügbarkeit zu überwachen.

1. Kontext & Problemstellung (Deutsch)

Während Logs die Frage beantworten "Was ist wann passiert?", beantworten Metriken die Frage "Was macht das System genau jetzt und im Zeitverlauf?". Für eine Echtzeit-Kommunikationsplattform wie Overtime müssen zentrale operative Signale – wie aktive WebSocket-Verbindungen, gleichzeitige Video-Calls, Anfrage-Latenzen und Fehlerraten – überwacht werden können, ohne dafür Log-Streams aufwendig parsen zu müssen.

Ein Pull-basiertes Metrik-System, das direkt im Anwendungsprozess läuft, wird gegenüber einem Push-Ansatz (bei dem die Anwendung Metriken aktiv an einen externen Dienst sendet) bevorzugt. Ein Push-Ansatz würde eine zusätzliche Netzwerk-Abhängigkeit im kritischen Schreibpfad (Write-Path) der Applikation einführen.

2. Architecture Decision Record (English)

ADR-012: Prometheus for Application Metrics

Status

Accepted

Context and Problem Statement

Logs answer "what happened and when." Metrics answer "what is the system doing right now and over time." For a real-time messaging and video-calling application, key operational signals — active WebSocket connections, concurrent video calls, request latency, error rate — must be observable without parsing log streams. A pull-based metrics system that runs in-process is preferable to pushing metrics from application code to an external service, which would introduce a write-path dependency.

Decision Drivers

  • Actionable Alerting: Metrics must be queryable for alerting (e.g., concurrent calls or WebSocket connections exceeding a safe threshold).
  • Zero Log Interference: The exposure mechanism must not interfere with application logging (scrape requests must not generate access log noise).
  • Accurate Multiprocessing: The multi-worker Gunicorn setup requires metrics to be correctly aggregated across all workers — naive per-worker reporting produces misleading totals.
  • Internal Accessibility: The /metrics endpoint must be reachable by an in-cluster Prometheus scraper without authentication, but must remain unexposed externally.

Considered Options

  • StatsD + Push Gateway — Application pushes counters to a sidecar. Requires maintaining a sidecar and a push gateway; metrics can be lost if the sidecar is temporarily unavailable.
  • OpenTelemetry SDK — Vendor-neutral but introduces a significantly larger dependency surface and requires an OTEL collector in the cluster. Too much setup overhead for a focused prototype.
  • **django-prometheus + prometheus_client** — Pull-based. The /metrics endpoint is served directly from the app process, Prometheus scrapes on its own schedule, and Gunicorn multiprocess support is natively built-in.

Decision Outcome

Chosen: django-prometheus for Django-level automatic instrumentation (request counts, latency histograms, DB query counts) combined with prometheus_client for custom business metrics.

Gunicorn Multiprocess Mode

Gunicorn forks multiple worker processes. Without coordination, each worker tracks its own counter values (e.g., a Prometheus scrape of worker A sees only worker A's counts). prometheus_client resolves this via a shared memory directory:

  • The PROMETHEUS_MULTIPROC_DIR environment variable points to /app/prometheus_multiproc (created during the builder stage of the Dockerfile).
  • Each worker writes its metric values to a file in this directory.
  • At scrape time, the prometheus_client collector seamlessly aggregates across all worker files.
Log Noise Suppression & Metric Naming

Prometheus scrapes /metrics every few seconds. Without filtering, each scrape generates a Django access log line, resulting in thousands of low-value logs obscuring real events. A NoiseFilter explicitly excludes /metrics from the django.request logger. The endpoint remains fully functional for scraping without polluting standard output.

All custom metrics use the overtime namespace prefix (e.g., overtime_concurrent_calls). The _total suffix on counters is a Prometheus convention automatically appended by prometheus_client.

Consequences & Trade-offs

Positive Consequences
  • Zero-Code Baselines: django-prometheus auto-instruments request duration histograms, status code counts, and database query counts with zero application code changes.
  • Simplified Architecture: Prometheus utilizes a pull model. The application has no knowledge of whether it is being scraped, removing outbound network dependencies.
  • Centralized Filtering: The NoiseFilter is shared between the metrics scrape path and the Kubernetes healthcheck path — a single mechanism suppresses all operational-polling noise.
Negative Consequences / Trade-offs
  • Strict Deployment Invariants: PROMETHEUS_MULTIPROC_DIR must exist and be writable before Gunicorn workers start. A missing or non-writable path causes a runtime crash.
  • Manual Mode Declaration: prometheus_client does not support Gauge multiprocess_mode="livesum" by default. Using the wrong mode produces silently incorrect aggregations; developers must declare this mode explicitly for every new Gauge.
  • Unauthenticated Endpoint: /metrics is unauthenticated. This is acceptable for an in-cluster scrape, but the ingress controller must be strictly configured to never expose this path externally.

3. Implementation & Code Snippets (English)

The implementation utilizes the prometheus_client to define custom metrics at the module level. They are updated directly at the relevant call sites without needing to pass a metrics client object through function signatures.

Custom Metrics Definition (api/views/demo.py)

The following snippet demonstrates the two primary metric types used in the Overtime backend: a Gauge for tracking active state (which goes up and down) and a Counter for tracking cumulative events (which only goes up).

from prometheus_client import Counter, Gauge

# Gauge: multiprocess_mode="livesum" sums the value across all Gunicorn workers so
# /metrics always shows the real total across the pod, not just one worker's slice.
concurrent_calls = Gauge(
    "concurrent_calls",
    "Number of calls currently being processed",
    namespace="overtime",
    multiprocess_mode="livesum",
)
# Usage when a video call starts:
# concurrent_calls.inc()
# Usage when a video call ends:
# concurrent_calls.dec()


# Counter: Each worker accumulates its own count; prometheus_client sums them 
# together automatically at scrape time (default behavior for Counters).
finished_calls = Counter(
    "finished_calls_total",
    "Total number of finished calls",
    namespace="overtime",
)
# Usage (Counters never decrease. Prometheus calculates rates over time itself):
# finished_calls.inc()

4. Compliance Einordnung (Deutsch)

Das detaillierte Mapping auf BSI IT-Grundschutz, CRA und CIS Benchmarks wird später ergänzt.