Logs and Metrics in Microservices: One Screen, Not Six Log Files

First, stop: what are you looking at?

The monolith had one log file and tail -f was enough. With twelve services (six of them in the chain that morning), the same habit became six tabs, and then a three-hour journey per bug. In the migration post I called this "invisibility"; this post is how we brought those three hours down to 25 minutes.

There are three signals, and they answer three different questions:

The order matters, and the reason for our three-hour journey was exactly that we had it backwards: you do not hunt for a spike by reading logs. Looking for "something odd" in six services' logs is a needle in a haystack; with metrics you first say "payment p95 tripled at 09:05", then open the log of that second. A one-minute job.

The setup: metrics to Prometheus, logs to Loki, both in Grafana

# each service exposes an endpoint; Prometheus comes and reads it every 15 s
GET /metrics

# counter: cumulative total since the process started (the error ratio in an
# incident window is read with rate(), not from this number)
http_requests_total{service="payment",route="/pay",status="200"}   48213
http_requests_total{service="payment",route="/pay",status="500"}     512

# histogram: cumulative buckets + sum + count
http_request_duration_seconds_bucket{service="payment",le="0.1"}   39880
http_request_duration_seconds_bucket{service="payment",le="0.3"}   47102
http_request_duration_seconds_bucket{service="payment",le="1"}     48180
http_request_duration_seconds_bucket{service="payment",le="2.5"}   48590   # upper buckets matter: if the largest
http_request_duration_seconds_bucket{service="payment",le="5"}     48700   # finite bucket were 1 s, a p95 of 1.8 s
http_request_duration_seconds_bucket{service="payment",le="10"}    48722   # could never be shown (it would cap at 1 s)
http_request_duration_seconds_bucket{service="payment",le="+Inf"}  48725   # required: quantiles do not work without it
http_request_duration_seconds_sum{service="payment"}                6417.2
http_request_duration_seconds_count{service="payment"}             48725

# the service writes one JSON line to stdout and does nothing else
{"ts":"2026-09-07T09:05:14.310+03:00","level":"error","service":"payment",
 "correlation_id":"c-7f3a","event_id":"E2","msg":"psp timeout","duration_ms":3012}

# the agent reads the container's stdout and ships it to Loki
# Loki indexes only the LABELS: service, level, env
# line content is not indexed -> cheap, but a query must start with a label

alloy convert --source-format=promtail --output=config.alloy promtail.yaml

The "indexes only the labels" line is why Loki is cheap: where Elasticsearch indexes every word, Loki indexes a handful of labels and stores the line itself compressed. For 40 GB of logs a day the Elasticsearch cluster was three servers; Loki was one server plus object storage. The price is in the same place: a query has to start with a label ({service="payment"}) and filter content afterwards. "Every line anywhere that mentions 4711" is expensive in Loki; "the payment service's last 10 minutes, mentioning 4711" is cheap.

Metrics: four golden signals, the rest is noise

Our first dashboard had 41 graphs: CPU, memory, GC time, thread counts, disk, network, JVM pools. Nobody looked at it, because none of them answered "what is the user experiencing right now". The second dashboard kept four graphs per service. The PromQL below can be copied as is; we had written two of them wrong at first, and the reason is underneath.

# error rate - BROKEN: the status label exists on both sides,
# Prometheus matches labels, so 500 is divided only by 500 -> always 1
rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m])

# p95 - INCOMPLETE: without sum by (le) you get one p95 per pod;
# not the service's p95 but N pod p95s, and the alert fires per pod
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

Three more traps, and we fell into all three:

# a developer added this to find out "which customer is slow":
http_request_duration_seconds{service="payment", user_id="184203", ...}

# every distinct user_id = a separate time series
# 2.4 million series -> Prometheus OOM -> every metric blind for 40 minutes

# prometheus.yml - on the scrape job
sample_limit: 20000      # a target returning more series than this is rejected
label_limit: 30          # at most 30 labels per metric
target_limit: 500

metric_relabel_configs:  # drop dangerous labels before ingestion
  - regex: "user_id|order_id|request_id"
    action: labeldrop

# look once a week: the busiest labels and metrics
curl -s localhost:9090/api/v1/status/tsdb | jq .data.seriesCountByMetricName

Logs: structure, not lines

The log lines inherited from the monolith looked like this:

2026-09-07 09:05:14 ERROR PaymentClient - timeout after 3012ms for order 4711 (retry 2)

{"ts":"2026-09-07T09:05:14.310+03:00","level":"error","service":"payment",
 "correlation_id":"c-7f3a","event_id":"E2","order_id":4711,
 "msg":"psp timeout","duration_ms":3012,"retry":2}

# LogQL: start with a label, trim with a line filter, then filter on fields
{service="payment"} |= "psp timeout" | json | duration_ms > 3000
{service=~"payment|shipping"} |= "c-7f3a" | json | correlation_id = "c-7f3a"

Four fields are mandatory on every line and the service skeleton sets them automatically: service, level, correlation_id, event_id. The developer writes only msg and their own fields. A line without those four is a line that does not exist. event_id is the identity of the domain event that caused the line to be written (OrderCreated = E1, say; one of the three IDs from the migration post): "which event was being processed when this line was written" is answered in one query.

One screen, one time axis

The layout is a reading order: what the user feels at the top (latency, errors), candidate causes in the middle (request rate, connection pool, consumer lag), evidence at the bottom (recent errors, active alerts). When the on-call engineer opens the dashboard in the morning their eye runs top to bottom, and by the third row the cause is usually obvious.

The real trick is not the graphs; it is the shared time axis. You select 09:05 on the latency graph; the log panel jumps to the same range; the service map shows who returned errors to whom in that minute, in red. Doing that by hand across three tools — copy the timestamp, switch tabs, paste, correct for the timezone — was half of the three hours.

The bridge from metrics to traces lives on the same screen: exemplars. When a histogram bucket is written, the trace_id of that sample is stored with it; click a point on the latency graph and Grafana takes you straight to that request's trace. The answer to "why is p95 1.8 s" is one click away: inside which service, in which call. Once Alloy delivers over OTLP the natural form of this is OpenTelemetry: trace_id and span_id land in the log line on their own, correlation_id gives way to them, and log → trace becomes one click too.

And the dashboard itself: its JSON and the alert rules live in git and are loaded by provisioning. A hand-edited dashboard turns into something nobody understands within a week; for the one in git it is clear who changed what, when and why.

Alerting: on what the user feels, not on causes

Back to that Monday morning. The CPU alert at 09:12 was correct — CPU really was at 68% — and completely useless. Users do not feel CPU; they feel latency and errors. Worse, that alert fired every Monday morning as traffic rose; the team had long since learned not to read it. An alert that fires constantly is the same as one that is muted.

# symptom threshold: p95 across the service (sum by le), firing if it lasts 5 minutes
- alert: PaymentLatency
  expr: histogram_quantile(0.95,
          sum by (le) (rate(http_request_duration_seconds_bucket{service="payment"}[5m]))) > 0.3
  for: 5m
  labels:  { severity: critical }
  annotations:
    summary: "payment p95 above 300 ms for 5 minutes"
    runbook: "https://wiki/runbook/payment-latency"   # the first three steps live here

# an alert built on rate() returns EMPTY when traffic drops to zero, and stays silent.
# these fire when the service is down entirely:
- alert: PaymentDown
  expr: up{job="payment"} == 0
  for: 2m
- alert: PaymentNoMetrics
  expr: absent(rate(http_requests_total{service="payment"}[5m]))
  for: 5m
- alert: PaymentTrafficDropped
  expr: sum(rate(http_requests_total{service="payment"}[5m]))
        < 0.3 * sum(rate(http_requests_total{service="payment"}[5m] offset 1w))
  for: 10m

# dead man's switch: ALWAYS fires. Alertmanager has a "tell me if this goes quiet" rule;
# if it goes quiet, Prometheus or the alerting chain itself is dead.
- alert: Watchdog
  expr: vector(1)
  labels: { severity: none }

An honesty note: the "symptom threshold" in the table is not an SLO alert; it is a better-chosen threshold. A real SLO alert is built on the error budget's burn rate: "fire if the budget is burning 14× too fast over the last hour and also over the last 5 minutes" — multiple windows, multiple rates. A static threshold knows how to stay quiet for a three-minute spike at 3am but never sees a 1% degradation that lasts all day; burn rate classifies both correctly. We moved there in month six; the numbers in this post belong to the symptom-threshold period.

The costs nobody budgeted: volume, retention, single point

Structured logs are good; logging everything is not. In month three daily log volume reached 40 GB, and half of it was one service writing every request twice at debug level. Three rules:

• info and above in production. debug is switched on only during an incident, for one service, for one hour — and a timer switches it back off.
• The happy path is one line. A request finishing successfully is one line; no step-by-step narration. Steps are metrics.
• Error lines are full (masked), success lines are short. An error line carries the context: the input masked, duration, retry count; a success line carries only IDs and duration.

Volume fell from 40 GB to 9 GB and search got faster; in Loki, query time is proportional to the data scanned.

The natural continuation of volume is retention. Prometheus's local retention defaults to 15 days; for "what happened last quarter" you need remote_write to Mimir or Thanos — we keep 13 months in Mimir. On the Loki side the real bill is object storage; without retention_period and the compactor, even 9 GB a day is 3 TB in a year. We keep 30 days hot, one year on a cheap tier, then delete.

And the single point: the Prometheus that went blind for 40 minutes with an OOM was a single instance. There are now two Prometheus instances in parallel scraping the same targets, both writing to Mimir; Grafana reads from Mimir. When one goes down, neither the data nor the dashboard disappears. It is one paragraph, but it is the reason the 40-minute blindness never happened again.

From the field: from 3 hours to 25 minutes

The last row is the least technical and the most useful. The dashboard is on the screen for the first minute of standup: what happened last night, which service is near its limit. Once the dashboard became something everyone looked at, the graphs evolved to match what people looked for; the 41-graph dashboard nobody looked at never evolved at all.

Checklist

Conclusion

Observability is not buying tools; it is building an arrangement where metrics say where to look, logs say what you are seeing, and the two sit side by side at the same second. Prometheus, Loki and Grafana make that cheap; but with the same three you can also build a 41-graph dashboard nobody looks at — we did. You can also build a dashboard that shows green on wrong PromQL — we did that too.

The sequence to remember: four signals first, with correct PromQL; then structured logs; then put both on one time axis; tie alerts to what the user feels, and make them fire when the service dies too; never use a label for an identifier, and put a guardrail behind that rule. The rest — the service map, the pretty gauges, the coloured panels — comes on its own once that arrangement exists, and is useless until it does.