Monitoring & Observability Master Class #

Engineering-Grade Reference Manual for Monitoring & Observability
A comprehensive guide to Prometheus, Grafana, logging, tracing, and production monitoring

---

Table of Contents #

1. Monitoring Fundamentals
2. Prometheus
3. Grafana
4. Logging with ELK Stack
5. Loki & Promtail
6. Distributed Tracing
7. Application Performance Monitoring
8. Alerting Strategies
9. SLIs, SLOs, and SLAs
10. Monitoring Kubernetes
11. Best Practices

---

Monitoring Fundamentals #

🔹 The Four Golden Signals #

Latency - Time to service a request Traffic - Demand on your system Errors - Rate of failed requests Saturation - How "full" your service is

🔹 Metrics vs Logs vs Traces #

METRICS (What's happening?)
- Numerical measurements over time
- CPU usage, request rate, error rate
- Aggregatable, queryable

LOGS (What happened?)
- Discrete events with context
- Error messages, audit trails
- Searchable, filterable

TRACES (Why did it happen?)
- Request flow through system
- Latency breakdown, dependencies
- Distributed system debugging

---

Prometheus #

🔹 Installation #

# Using Docker
docker run -d \
  --name prometheus \
  -p 9090:9090 \
  -v $(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml \
  prom/prometheus

# Kubernetes with Helm
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus prometheus-community/kube-prometheus-stack

🔹 Configuration #

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets:
          - alertmanager:9093

rule_files:
  - "alerts.yml"

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
  
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']
  
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__

🔹 PromQL Queries #

# CPU usage
rate(process_cpu_seconds_total[5m])

# Memory usage
process_resident_memory_bytes / 1024 / 1024

# HTTP request rate
rate(http_requests_total[5m])

# Error rate
rate(http_requests_total{status=~"5.."}[5m])

# 95th percentile latency
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

# Requests per second by endpoint
sum(rate(http_requests_total[5m])) by (endpoint)

# Top 5 endpoints by traffic
topk(5, sum(rate(http_requests_total[5m])) by (endpoint))

# Availability (uptime)
avg_over_time(up[24h])

🔹 Application Instrumentation #

Node.js Example:

const client = require('prom-client');
const express = require('express');

// Create a Registry
const register = new client.Registry();

// Add default metrics
client.collectDefaultMetrics({ register });

// Custom counter
const httpRequestsTotal = new client.Counter({
  name: 'http_requests_total',
  help: 'Total HTTP requests',
  labelNames: ['method', 'route', 'status'],
  registers: [register]
});

// Custom histogram
const httpRequestDuration = new client.Histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request duration',
  labelNames: ['method', 'route', 'status'],
  buckets: [0.1, 0.5, 1, 2, 5],
  registers: [register]
});

const app = express();

// Middleware to track metrics
app.use((req, res, next) => {
  const start = Date.now();
  
  res.on('finish', () => {
    const duration = (Date.now() - start) / 1000;
    httpRequestsTotal.inc({
      method: req.method,
      route: req.route?.path || req.path,
      status: res.statusCode
    });
    httpRequestDuration.observe({
      method: req.method,
      route: req.route?.path || req.path,
      status: res.statusCode
    }, duration);
  });
  
  next();
});

// Metrics endpoint
app.get('/metrics', async (req, res) => {
  res.set('Content-Type', register.contentType);
  res.end(await register.metrics());
});

---

Grafana #

🔹 Installation #

# Docker
docker run -d \
  --name=grafana \
  -p 3000:3000 \
  grafana/grafana

# Kubernetes
helm repo add grafana https://grafana.github.io/helm-charts
helm install grafana grafana/grafana

🔹 Dashboard JSON #

{
  "dashboard": {
    "title": "Application Metrics",
    "panels": [
      {
        "title": "Request Rate",
        "targets": [
          {
            "expr": "sum(rate(http_requests_total[5m])) by (route)",
            "legendFormat": "{{route}}"
          }
        ],
        "type": "graph"
      },
      {
        "title": "Error Rate",
        "targets": [
          {
            "expr": "sum(rate(http_requests_total{status=~\"5..\"}[5m]))",
            "legendFormat": "5xx errors"
          }
        ],
        "type": "graph"
      },
      {
        "title": "P95 Latency",
        "targets": [
          {
            "expr": "histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))",
            "legendFormat": "p95"
          }
        ],
        "type": "graph"
      }
    ]
  }
}

---

Logging with ELK Stack #

🔹 Docker Compose Setup #

version: '3'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
    environment:
      - discovery.type=single-node
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ports:
      - "9200:9200"
  
  logstash:
    image: docker.elastic.co/logstash/logstash:8.11.0
    volumes:
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf
    ports:
      - "5000:5000"
    depends_on:
      - elasticsearch
  
  kibana:
    image: docker.elastic.co/kibana/kibana:8.11.0
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch

🔹 Logstash Configuration #

# logstash.conf
input {
  tcp {
    port => 5000
    codec => json
  }
}

filter {
  if [level] == "error" {
    mutate {
      add_tag => ["error"]
    }
  }
  
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}" }
  }
  
  date {
    match => [ "timestamp", "ISO8601" ]
  }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    index => "logs-%{+YYYY.MM.dd}"
  }
}

---

Loki & Promtail #

🔹 Loki Configuration #

# loki-config.yaml
auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
  chunk_idle_period: 5m
  chunk_retain_period: 30s

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 168h

storage_config:
  boltdb:
    directory: /tmp/loki/index
  filesystem:
    directory: /tmp/loki/chunks

limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h

🔹 Promtail Configuration #

# promtail-config.yaml
server:
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          __path__: /var/log/*log
  
  - job_name: containers
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        target_label: 'container'

🔹 LogQL Queries #

# All logs from a container
{container="myapp"}

# Error logs
{container="myapp"} |= "error"

# JSON parsing
{container="myapp"} | json | level="error"

# Rate of errors
rate({container="myapp"} |= "error" [5m])

# Top 10 error messages
topk(10, sum by (message) (rate({container="myapp"} |= "error" [5m])))

---

Distributed Tracing #

🔹 Jaeger Setup #

# docker-compose.yml
version: '3'
services:
  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - "5775:5775/udp"
      - "6831:6831/udp"
      - "6832:6832/udp"
      - "5778:5778"
      - "16686:16686"
      - "14268:14268"
      - "9411:9411"
    environment:
      - COLLECTOR_ZIPKIN_HTTP_PORT=9411

🔹 Application Instrumentation #

// Node.js with OpenTelemetry
const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
const { JaegerExporter } = require('@opentelemetry/exporter-jaeger');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');

const provider = new NodeTracerProvider({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: 'my-service',
  }),
});

const exporter = new JaegerExporter({
  endpoint: 'http://jaeger:14268/api/traces',
});

provider.addSpanProcessor(new BatchSpanProcessor(exporter));
provider.register();

// Create spans
const tracer = provider.getTracer('my-app');

app.get('/api/users', async (req, res) => {
  const span = tracer.startSpan('get-users');
  
  try {
    const users = await db.query('SELECT * FROM users');
    span.setStatus({ code: SpanStatusCode.OK });
    res.json(users);
  } catch (error) {
    span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });
    res.status(500).json({ error: error.message });
  } finally {
    span.end();
  }
});

---

Application Performance Monitoring #

🔹 Key Metrics to Monitor #

Application Metrics:
  - Request rate (requests/second)
  - Error rate (errors/second)
  - Response time (p50, p95, p99)
  - Throughput (bytes/second)

System Metrics:
  - CPU usage (%)
  - Memory usage (MB/GB)
  - Disk I/O (IOPS, throughput)
  - Network I/O (packets, bandwidth)

Business Metrics:
  - User signups
  - Transactions completed
  - Revenue generated
  - Active users

---

Alerting Strategies #

🔹 Alert Rules #

# alerts.yml
groups:
  - name: application
    interval: 30s
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High error rate detected"
          description: "Error rate is {{ $value }} errors/sec"
      
      - alert: HighLatency
        expr: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High latency detected"
          description: "P95 latency is {{ $value }}s"
      
      - alert: ServiceDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Service is down"
          description: "{{ $labels.instance }} is down"

🔹 Alertmanager Configuration #

# alertmanager.yml
global:
  resolve_timeout: 5m
  slack_api_url: 'https://hooks.slack.com/services/YOUR/WEBHOOK/URL'

route:
  group_by: ['alertname', 'cluster']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 12h
  receiver: 'slack'
  routes:
    - match:
        severity: critical
      receiver: 'pagerduty'
    - match:
        severity: warning
      receiver: 'slack'

receivers:
  - name: 'slack'
    slack_configs:
      - channel: '#alerts'
        text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'
  
  - name: 'pagerduty'
    pagerduty_configs:
      - service_key: 'YOUR_PAGERDUTY_KEY'

---

SLIs, SLOs, and SLAs #

🔹 Definitions #

SLI (Service Level Indicator)

• Quantitative measure of service level
• Examples: latency, error rate, availability

SLO (Service Level Objective)

• Target value for SLI
• Example: 99.9% availability

SLA (Service Level Agreement)

• Contract with consequences
• Example: 99.9% uptime or refund

🔹 Example SLOs #

Availability SLO:
  Target: 99.9% (43.2 minutes downtime/month)
  Measurement: (successful requests / total requests) * 100

Latency SLO:
  Target: 95% of requests < 200ms
  Measurement: histogram_quantile(0.95, http_request_duration_seconds)

Error Budget:
  Allowed failures: 0.1% (99.9% SLO)
  Monthly budget: 43.2 minutes
  Remaining: Track in real-time

---

Monitoring Kubernetes #

🔹 Metrics Server #

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

# View node metrics
kubectl top nodes

# View pod metrics
kubectl top pods -n production

🔹 ServiceMonitor #

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: myapp
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: myapp
  endpoints:
    - port: metrics
      interval: 30s
      path: /metrics

---

Best Practices #

✅ Monitoring Best Practices #

1. Monitor the Four Golden Signals
2. Set meaningful alerts - Avoid alert fatigue
3. Use labels effectively - For filtering and grouping
4. Retain metrics appropriately - Balance cost vs value
5. Document your dashboards - Add descriptions
6. Test your alerts - Ensure they fire correctly
7. Monitor your monitoring - Meta-monitoring
8. Use recording rules - Pre-compute expensive queries

✅ Dashboard Design #

1. Start with overview - High-level health
2. Drill-down capability - From overview to details
3. Use consistent colors - Green=good, Red=bad
4. Show trends - Not just current values
5. Include context - Annotations for deployments
6. Optimize query performance - Use recording rules

✅ Alert Design #

1. Alert on symptoms, not causes
2. Make alerts actionable
3. Include runbooks in annotations
4. Set appropriate thresholds
5. Use severity levels
6. Avoid noisy alerts

---

🎓 Monitoring & Observability Master Class Complete

This guide covers essential monitoring and observability tools and practices for production systems.