DevOps Troubleshooting Guide #

Common Problems and Solutions
Quick fixes for frequent DevOps issues

---

Docker Troubleshooting #

❌ Container Won't Start #

Problem: Container exits immediately

docker logs container-id

Solutions:

• Check logs for errors
• Verify CMD/ENTRYPOINT in Dockerfile
• Ensure application doesn't exit immediately
• Check for missing dependencies

❌ Port Already in Use #

Problem: bind: address already in use

Solution:

# Find process using port
lsof -i :8080
# or
netstat -tulpn | grep 8080

# Kill process
kill -9 PID

# Use different port
docker run -p 8081:80 nginx

❌ Out of Disk Space #

Problem: no space left on device

Solution:

# Clean up
docker system prune -a --volumes

# Check disk usage
docker system df

# Remove unused images
docker image prune -a

# Remove unused volumes
docker volume prune

---

Kubernetes Troubleshooting #

❌ Pod Stuck in Pending #

Problem: Pod won't schedule

Diagnosis:

kubectl describe pod pod-name
kubectl get events --sort-by='.lastTimestamp'

Common Causes:

• Insufficient resources (CPU/memory)
• Node selector mismatch
• PersistentVolume not available
• Image pull errors

Solutions:

# Check node resources
kubectl top nodes

# Check pod resource requests
kubectl describe pod pod-name | grep -A 5 "Requests"

# Scale down other deployments
kubectl scale deployment other-app --replicas=0

❌ CrashLoopBackOff #

Problem: Pod keeps restarting

Diagnosis:

kubectl logs pod-name
kubectl logs pod-name --previous
kubectl describe pod pod-name

Common Causes:

• Application crashes on startup
• Liveness probe failing
• Missing configuration/secrets
• Insufficient permissions

Solutions:

# Check logs
kubectl logs pod-name --previous

# Exec into pod (if possible)
kubectl exec -it pod-name -- sh

# Check events
kubectl get events --field-selector involvedObject.name=pod-name

❌ ImagePullBackOff #

Problem: Can't pull container image

Solutions:

# Check image name
kubectl describe pod pod-name | grep Image

# Check image pull secrets
kubectl get secrets

# Create image pull secret
kubectl create secret docker-registry regcred \
  --docker-server=registry.example.com \
  --docker-username=user \
  --docker-password=pass

---

Git Troubleshooting #

❌ Merge Conflicts #

Problem: Conflicts during merge

Solution:

# See conflicted files
git status

# Open and resolve conflicts manually
# Look for <<<<<<< HEAD markers

# After resolving
git add resolved-file.txt
git commit

# Abort merge if needed
git merge --abort

❌ Detached HEAD #

Problem: Not on any branch

Solution:

# Create branch from current state
git checkout -b recovery-branch

# Or return to previous branch
git checkout main

❌ Accidentally Committed Secrets #

Problem: Pushed sensitive data

Solution:

# Remove file from history
git filter-branch --force --index-filter \
  "git rm --cached --ignore-unmatch secrets.txt" \
  --prune-empty --tag-name-filter cat -- --all

# Force push
git push origin --force --all

# Rotate compromised secrets immediately!

---

Terraform Troubleshooting #

❌ State Lock Error #

Problem: State is locked

Solution:

# Force unlock (use carefully!)
terraform force-unlock LOCK_ID

# Check who has lock
# Look in state backend (S3, Azure Storage, etc.)

❌ Resource Already Exists #

Problem: Resource exists but not in state

Solution:

# Import existing resource
terraform import aws_instance.example i-1234567890abcdef0

# Or remove from state and let Terraform recreate
terraform state rm aws_instance.example

---

Ansible Troubleshooting #

❌ Connection Refused #

Problem: Can't connect to hosts

Diagnosis:

ansible all -m ping -vvv

Solutions:

# Check SSH connectivity
ssh user@host

# Verify inventory
ansible-inventory --list

# Check SSH key
ssh-add -l

# Use password authentication
ansible all -m ping --ask-pass

❌ Permission Denied #

Problem: Can't perform privileged operations

Solution:

# Use become
ansible-playbook playbook.yml --become

# Ask for become password
ansible-playbook playbook.yml --become --ask-become-pass

---

CI/CD Troubleshooting #

❌ Pipeline Fails Intermittently #

Problem: Flaky tests or network issues

Solution:

# Add retry logic (GitHub Actions)
- name: Flaky test
  uses: nick-invision/retry@v2
  with:
    timeout_minutes: 10
    max_attempts: 3
    command: npm run test:e2e

❌ Out of Disk Space in CI #

Problem: Runner runs out of space

Solution:

# Clean up before build
- name: Free disk space
  run: |
    docker system prune -af
    sudo rm -rf /usr/share/dotnet
    sudo rm -rf /opt/ghc

---

Database Troubleshooting #

❌ PostgreSQL Connection Refused #

Problem: Can't connect to database

Solutions:

# Check if PostgreSQL is running
sudo systemctl status postgresql

# Check listen address
sudo grep listen_addresses /etc/postgresql/*/main/postgresql.conf

# Check pg_hba.conf for access rules
sudo cat /etc/postgresql/*/main/pg_hba.conf

# Restart PostgreSQL
sudo systemctl restart postgresql

❌ Slow Queries #

Problem: Database performance issues

Diagnosis:

-- PostgreSQL: Find slow queries
SELECT pid, now() - query_start AS duration, query
FROM pg_stat_activity
WHERE state = 'active'
ORDER BY duration DESC;

-- Analyze query
EXPLAIN ANALYZE SELECT * FROM users WHERE email = 'test@example.com';

Solutions:

-- Create index
CREATE INDEX idx_email ON users(email);

-- Vacuum database
VACUUM ANALYZE;

---

Monitoring Troubleshooting #

❌ Prometheus Not Scraping Targets #

Problem: Targets show as down

Diagnosis:

# Check Prometheus targets page
http://prometheus:9090/targets

# Check if metrics endpoint is accessible
curl http://app:8080/metrics

Solutions:

• Verify network connectivity
• Check firewall rules
• Verify metrics endpoint path
• Check service discovery configuration

---

Quick Diagnostic Commands #

# System health
top
htop
df -h
free -h
iostat
vmstat

# Network
ping host
traceroute host
netstat -tulpn
ss -tulpn
tcpdump -i eth0

# Logs
journalctl -u service -f
tail -f /var/log/syslog
dmesg | tail

# Docker
docker logs container-id
docker inspect container-id
docker stats

# Kubernetes
kubectl get events --sort-by='.lastTimestamp'
kubectl top nodes
kubectl top pods
kubectl describe pod pod-name

---

🎓 DevOps Troubleshooting Guide Complete