Observability and Monitoring Architecture

The Three Pillars of Observability

Observability rests on three data types: metrics, logs, and traces. Metrics are numeric measurements over time -- request rate, error rate, latency percentiles, CPU utilization. Logs are discrete events with structured context -- a user action, an error with a stack trace, a configuration change. Traces follow a request across service boundaries, showing the timing and dependencies of each step in a distributed workflow.

Each pillar answers different questions. Metrics answer 'what is happening' -- error rate is spiking, latency is increasing, queue depth is growing. Logs answer 'what happened specifically' -- this request failed with this error at this time. Traces answer 'why is this slow' -- this request spent 2 seconds in the database query, 500ms in the external API call, and 100ms in serialization. A complete observability stack requires all three because no single pillar provides a full picture.

The shift from monitoring to observability reflects the shift from monoliths to distributed systems. Monitoring a monolith with CPU, memory, and error rate dashboards works because the system has a single failure mode. A distributed system with 20 services has hundreds of failure modes, many of which are emergent -- they arise from interactions between services rather than failures within individual services. Observability provides the data needed to investigate these emergent problems without knowing in advance what questions to ask.

Metrics Collection and Storage

Prometheus has become the standard for metrics collection in cloud-native environments. It scrapes metrics endpoints on a configured interval, stores them in a time-series database, and provides PromQL for querying. Grafana provides the visualization layer. This open-source stack is free, well-documented, and supported by a large ecosystem of exporters for common infrastructure components. For teams that prefer managed services, Datadog, New Relic, and Grafana Cloud provide equivalent capabilities without operational overhead.

Define metrics using the RED method for services and the USE method for infrastructure. RED -- Rate, Errors, Duration -- covers the three essential metrics for any request-serving service. USE -- Utilization, Saturation, Errors -- covers infrastructure resources like CPU, memory, disk, and network. Brendan Gregg developed USE and documents it extensively. These two frameworks ensure comprehensive coverage without overwhelming the team with hundreds of metrics.

Use histograms rather than averages for latency metrics. An average latency of 200ms might hide the fact that 99% of requests complete in 100ms while 1% take 10 seconds. The 50th, 95th, and 99th percentiles reveal the distribution. Prometheus histograms and summary types support percentile calculation natively. Set alerting thresholds on the 95th or 99th percentile rather than the average to catch long-tail latency issues that affect a minority of users but indicate real problems.

Structured Logging at Scale

Unstructured log messages -- plain text strings like 'Error processing order' -- are nearly useless at scale because they cannot be queried efficiently. Structured logging emits log entries as JSON objects with consistent fields: timestamp, severity, service name, trace ID, user ID, and error details. This structure enables filtering, aggregation, and correlation across millions of log entries per hour.

Centralize logs using the ELK stack (Elasticsearch, Logstash, Kibana), Grafana Loki, or a managed service like Datadog Logs or Splunk. Log aggregation across services is essential for troubleshooting distributed systems -- a single user request might generate log entries in five different services, and correlating them requires a shared trace ID that appears in every entry. OpenTelemetry provides standardized instrumentation that propagates trace context across service boundaries.

Control log volume to manage costs. Logging every request at debug level generates terabytes of data that costs thousands per month to store and index. Log at INFO level for normal operations and DEBUG level only when actively troubleshooting. Use sampling for high-volume endpoints -- log 1% of successful requests but 100% of errors. Implement log retention policies: keep detailed logs for 7-14 days and aggregated metrics for 13 months. This tiered approach balances troubleshooting capability with storage costs.

Distributed Tracing Implementation

Distributed tracing follows a request through every service it touches, recording timing information at each step. OpenTelemetry is the industry standard for trace instrumentation -- it provides SDKs for every major programming language and exports data to any trace backend. Jaeger and Zipkin are popular open-source trace backends. Managed services like Datadog APM, Honeycomb, and Lightstep provide trace collection with advanced analysis capabilities.

Instrument at service boundaries first -- incoming HTTP requests, outgoing HTTP requests, database queries, and message queue operations. Most OpenTelemetry SDKs provide automatic instrumentation for popular frameworks and libraries, requiring minimal code changes. Manual instrumentation adds spans for business logic -- a span around the payment processing step, the inventory check, or the recommendation calculation. Start with automatic instrumentation and add manual spans where additional detail is needed for troubleshooting.

Trace sampling is necessary at scale. Recording every trace for every request generates enormous data volumes. Head-based sampling decides at the start of a request whether to record it -- a 10% sampling rate records one in ten requests. Tail-based sampling makes the decision after the request completes, which allows capturing 100% of error and slow traces while sampling normal traces at a lower rate. Tail-based sampling produces more useful data but requires buffering complete traces before making the sampling decision.

Alerting Strategy and On-Call Practice

Alert on symptoms, not causes. A symptom is something users experience: high error rate, slow response time, failed transactions. A cause is an internal metric: high CPU utilization, low disk space, database connection pool exhaustion. Alert on symptoms because they represent actual impact. Investigate causes as part of troubleshooting. An alert on high CPU that does not affect user experience creates noise; an alert on high error rate that users experience creates actionable urgency.

Every alert should have a runbook -- a documented procedure for investigating and resolving the issue. The runbook should include: how to verify the alert is genuine, what to check first, common root causes and their fixes, and escalation procedures if the on-call engineer cannot resolve the issue. Runbooks reduce mean time to resolution by giving engineers a starting point rather than requiring them to diagnose from scratch at 3 AM.

Measure alert quality using the signal-to-noise ratio. If more than 30% of alerts are false positives or do not require action, the alerting configuration needs tuning. Alert fatigue -- where engineers start ignoring alerts because most are not meaningful -- is the most dangerous outcome of noisy alerting. Review every alert quarterly: was it actionable? Did it represent real user impact? Could it be deduplicated or suppressed during known maintenance windows? PagerDuty's operations health report provides benchmarks for alert volume and response times by team size and industry.