Site reliability engineer Applications that stay up, and changes that can be undone.
For teams that hear about problems from their users, or whose deploys feel like a held breath: monitors, SLOs and alerts in code beside the infrastructure, a way back for every change, and applications kept healthy in production together with their developers.
Four parts of reliability work.
Reliability kept in the same code and the same reviews as everything else.
-
Observability as code
Monitors and SLOs, reviewed like code.
Monitors, SLOs and log pipelines in Datadog, CloudWatch alarms and dashboards, and DNS health checks, all in Terraform: alerts defined next to the service they watch, and changed through a pull request.
-
Change management
Every change planned with a way back.
A risky change gets a written plan: what moves, in what order, how we will know it worked and how to roll it back. Infrastructure changes show their plan in review before they apply, and a release that goes wrong is rolled back rather than patched live.
-
Incident triage
Triage that starts from the evidence.
When something breaks: the logs, the metrics and the latest changes first, then the fix, then a note on what to watch next time. Cloud provider health notices get the same treatment.
-
With the developers
Running the application, not only the servers.
I work with the developers on what their applications need in production: queues, caches, search, timeouts and health checks, whatever language the application is written in. Dependencies stay current with Dependabot, so a security fix arrives as a reviewed pull request.
The tools behind reliability work.
Monitoring and alerting
- CloudWatch (AWS monitoring, logs and alarms)
- Structured logging
- Log review
- Monitoring as code
- SLO (Service level objective)s
- Log pipelines
- Health checks
- Alerting
- Datadog (Monitoring and logging service)
Changes and releases
- Change management
- Rollback plans
- Incident triage
- Gated Terraform plan and apply
- Release automation
- Terraform
DNS, certificates and the edge
- Route 53 (Amazon's DNS service)
- ACM (AWS Certificate Manager)
- CloudFront (Amazon's content delivery network)
- CloudFront Functions (Small functions that run at the CloudFront edge)
- WAF (AWS WAF, the web application firewall)
Running applications
- ECS (Amazon Elastic Container Service)
- Docker
- SQS (Amazon Simple Queue Service)
- SNS (Amazon Simple Notification Service)
- ElastiCache (Managed Redis on AWS)
- Nginx
- Gunicorn (Python web server for applications)
- Supporting development teams
Keeping applications patched
- Dependabot (Opens pull requests that keep dependencies up to date)
- Trivy (Vulnerability scanner for container images and infrastructure code)
- pip-audit (Checks Python dependencies for known vulnerabilities)
- Bandit (Security linter for Python)
- npm audit (Checks JavaScript dependencies for known vulnerabilities)
Blue marks what I work with now.
The same work, seen from two other sides.
Need a site reliability engineer?
Tell me what keeps breaking, or what you are afraid to change.