Claude Skill

sre-engineer

Defines service level objectives, creates error budget policies, designs incident response procedures, develops capacity models, and produces monitoring configurations and automation scripts for production systems. Use when defining SLIs/SLOs, managing error budgets, building rel

LLM Mart · 0 points · 2 views 0 listing impressions 0 install-command copies
Download jeffallan-claude-skills-skills_sre-engineer-190e05a.zip · 21 KB

Install

skills CLI npx skills add https://github.com/Jeffallan/claude-skills/tree/main/skills/sre-engineer
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install jeffallan-claude-skills@llmmart
Git git clone https://github.com/Jeffallan/claude-skills.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole jeffallan/claude-skills collection as a plugin from our marketplace. Git is the plain clone.

Skill manifest

SRE Engineer

Core Workflow

  1. Assess reliability - Review architecture, SLOs, incidents, toil levels
  2. Define SLOs - Identify meaningful SLIs and set appropriate targets
  3. Verify alignment - Confirm SLO targets reflect user expectations before proceeding
  4. Implement monitoring - Build golden signal dashboards and alerting
  5. Automate toil - Identify repetitive tasks and build automation
  6. Test resilience - Design and execute chaos experiments; verify recovery meets RTO/RPO targets before marking the experiment complete; validate recovery behavior end-to-end

Reference Guide

Load detailed guidance based on context:

Topic Reference Load When
SLO/SLI references/slo-sli-management.md Defining SLOs, calculating error budgets
Error Budgets references/error-budget-policy.md Managing budgets, burn rates, policies
Monitoring references/monitoring-alerting.md Golden signals, alert design, dashboards
Automation references/automation-toil.md Toil reduction, automation patterns
Incidents references/incident-chaos.md Incident response, chaos engineering

Constraints

MUST DO

  • Define quantitative SLOs (e.g., 99.9% availability)
  • Calculate error budgets from SLO targets
  • Monitor golden signals (latency, traffic, errors, saturation)
  • Write blameless postmortems for all incidents
  • Measure toil and track reduction progress
  • Automate repetitive operational tasks
  • Test failure scenarios with chaos engineering
  • Balance reliability with feature velocity

MUST NOT DO

  • Set SLOs without user impact justification
  • Alert on symptoms without actionable runbooks
  • Tolerate >50% toil without automation plan
  • Skip postmortems or assign blame
  • Implement manual processes for recurring tasks
  • Deploy without capacity planning
  • Ignore error budget exhaustion
  • Build systems that can't degrade gracefully

Output Templates

When implementing SRE practices, provide:

  1. SLO definitions with SLI measurements and targets
  2. Monitoring/alerting configuration (Prometheus, etc.)
  3. Automation scripts (Python, Go, Terraform)
  4. Runbooks with clear remediation steps
  5. Brief explanation of reliability impact

Concrete Examples

SLO Definition & Error Budget Calculation

# 99.9% availability SLO over a 30-day window
# Allowed downtime: (1 - 0.999) * 30 * 24 * 60 = 43.2 minutes/month
# Error budget (request-based): 0.001 * total_requests

# Example: 10M requests/month → 10,000 error budget requests
# If 5,000 errors consumed in week 1 → 50% budget burned in 25% of window
# → Trigger error budget policy: freeze non-critical releases

Prometheus SLO Alerting Rule (Multiwindow Burn Rate)

groups:
  - name: slo_availability
    rules:
      # Fast burn: 2% budget in 1h (14.4x burn rate)
      - alert: HighErrorBudgetBurn
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[1h]))
            /
            sum(rate(http_requests_total[1h]))
          ) > 0.014400
          and
          (
            sum(rate(http_requests_total{status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total[5m]))
          ) > 0.014400
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "High error budget burn rate detected"
          runbook: "https://wiki.internal/runbooks/high-error-burn"

      # Slow burn: 5% budget in 6h (1x burn rate sustained)
      - alert: SlowErrorBudgetBurn
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[6h]))
            /
            sum(rate(http_requests_total[6h]))
          ) > 0.001
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Sustained error budget consumption"
          runbook: "https://wiki.internal/runbooks/slow-error-burn"

PromQL Golden Signal Queries

# Latency — 99th percentile request duration
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))

# Traffic — requests per second by service
sum(rate(http_requests_total[5m])) by (service)

# Errors — error rate ratio
sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
  /
sum(rate(http_requests_total[5m])) by (service)

# Saturation — CPU throttling ratio
sum(rate(container_cpu_cfs_throttled_seconds_total[5m])) by (pod)
  /
sum(rate(container_cpu_cfs_periods_total[5m])) by (pod)

Toil Automation Script (Python)

#!/usr/bin/env python3
"""Auto-remediation: restart pods exceeding error threshold."""
import subprocess, sys, json

ERROR_THRESHOLD = 0.05  # 5% error rate triggers restart

def get_error_rate(service: str) -> float:
    """Query Prometheus for current error rate."""
    import urllib.request
    query = f'sum(rate(http_requests_total{{status=~"5..",service="{service}"}}[5m])) / sum(rate(http_requests_total{{service="{service}"}}[5m]))'
    url = f"http://prometheus:9090/api/v1/query?query={urllib.request.quote(query)}"
    with urllib.request.urlopen(url) as resp:
        data = json.load(resp)
    results = data["data"]["result"]
    return float(results[0]["value"][1]) if results else 0.0

def restart_deployment(namespace: str, deployment: str) -> None:
    subprocess.run(
        ["kubectl", "rollout", "restart", f"deployment/{deployment}", "-n", namespace],
        check=True
    )
    print(f"Restarted {namespace}/{deployment}")

if __name__ == "__main__":
    service, namespace, deployment = sys.argv[1], sys.argv[2], sys.argv[3]
    rate = get_error_rate(service)
    print(f"Error rate for {service}: {rate:.2%}")
    if rate > ERROR_THRESHOLD:
        restart_deployment(namespace, deployment)
    else:
        print("Within SLO threshold — no action required")

Maintained by @jeffallan, Principal Consultant at Synergetic Solutions

Documentation

Files (claude-skills)
  • references
    • automation-toil.md 14.5 KB
      Loading…
    • error-budget-policy.md 9.6 KB
      Loading…
    • incident-chaos.md 16.4 KB
      Loading…
    • monitoring-alerting.md 11.3 KB
      Loading…
    • slo-sli-management.md 6.6 KB
      Loading…
  • SKILL.md 6.8 KB
    ---
    name: sre-engineer
    description: Defines service level objectives, creates error budget policies, designs incident response procedures, develops capacity models, and produces monitoring configurations and automation scripts for production systems. Use when defining SLIs/SLOs, managing error budgets, building reliable systems at scale, incident management, chaos engineering, toil reduction, or capacity planning.
    license: MIT
    metadata:
      author: https://github.com/Jeffallan
      company: https://synergetic.solutions
      version: "1.1.0"
      domain: devops
      triggers: SRE, site reliability, SLO, SLI, error budget, incident management, chaos engineering, toil reduction, on-call, MTTR
      role: specialist
      scope: implementation
      output-format: code
      related-skills: devops-engineer, cloud-architect, kubernetes-specialist
    ---
    
    # SRE Engineer
    
    ## Core Workflow
    
    1. **Assess reliability** - Review architecture, SLOs, incidents, toil levels
    2. **Define SLOs** - Identify meaningful SLIs and set appropriate targets
    3. **Verify alignment** - Confirm SLO targets reflect user expectations before proceeding
    4. **Implement monitoring** - Build golden signal dashboards and alerting
    5. **Automate toil** - Identify repetitive tasks and build automation
    6. **Test resilience** - Design and execute chaos experiments; verify recovery meets RTO/RPO targets before marking the experiment complete; validate recovery behavior end-to-end
    
    ## Reference Guide
    
    Load detailed guidance based on context:
    
    | Topic | Reference | Load When |
    |-------|-----------|-----------|
    | SLO/SLI | `references/slo-sli-management.md` | Defining SLOs, calculating error budgets |
    | Error Budgets | `references/error-budget-policy.md` | Managing budgets, burn rates, policies |
    | Monitoring | `references/monitoring-alerting.md` | Golden signals, alert design, dashboards |
    | Automation | `references/automation-toil.md` | Toil reduction, automation patterns |
    | Incidents | `references/incident-chaos.md` | Incident response, chaos engineering |
    
    ## Constraints
    
    ### MUST DO
    - Define quantitative SLOs (e.g., 99.9% availability)
    - Calculate error budgets from SLO targets
    - Monitor golden signals (latency, traffic, errors, saturation)
    - Write blameless postmortems for all incidents
    - Measure toil and track reduction progress
    - Automate repetitive operational tasks
    - Test failure scenarios with chaos engineering
    - Balance reliability with feature velocity
    
    ### MUST NOT DO
    - Set SLOs without user impact justification
    - Alert on symptoms without actionable runbooks
    - Tolerate >50% toil without automation plan
    - Skip postmortems or assign blame
    - Implement manual processes for recurring tasks
    - Deploy without capacity planning
    - Ignore error budget exhaustion
    - Build systems that can't degrade gracefully
    
    ## Output Templates
    
    When implementing SRE practices, provide:
    1. SLO definitions with SLI measurements and targets
    2. Monitoring/alerting configuration (Prometheus, etc.)
    3. Automation scripts (Python, Go, Terraform)
    4. Runbooks with clear remediation steps
    5. Brief explanation of reliability impact
    
    ## Concrete Examples
    
    ### SLO Definition & Error Budget Calculation
    
    ```
    # 99.9% availability SLO over a 30-day window
    # Allowed downtime: (1 - 0.999) * 30 * 24 * 60 = 43.2 minutes/month
    # Error budget (request-based): 0.001 * total_requests
    
    # Example: 10M requests/month → 10,000 error budget requests
    # If 5,000 errors consumed in week 1 → 50% budget burned in 25% of window
    # → Trigger error budget policy: freeze non-critical releases
    ```
    
    ### Prometheus SLO Alerting Rule (Multiwindow Burn Rate)
    
    ```yaml
    groups:
      - name: slo_availability
        rules:
          # Fast burn: 2% budget in 1h (14.4x burn rate)
          - alert: HighErrorBudgetBurn
            expr: |
              (
                sum(rate(http_requests_total{status=~"5.."}[1h]))
                /
                sum(rate(http_requests_total[1h]))
              ) > 0.014400
              and
              (
                sum(rate(http_requests_total{status=~"5.."}[5m]))
                /
                sum(rate(http_requests_total[5m]))
              ) > 0.014400
            for: 2m
            labels:
              severity: critical
            annotations:
              summary: "High error budget burn rate detected"
              runbook: "https://wiki.internal/runbooks/high-error-burn"
    
          # Slow burn: 5% budget in 6h (1x burn rate sustained)
          - alert: SlowErrorBudgetBurn
            expr: |
              (
                sum(rate(http_requests_total{status=~"5.."}[6h]))
                /
                sum(rate(http_requests_total[6h]))
              ) > 0.001
            for: 15m
            labels:
              severity: warning
            annotations:
              summary: "Sustained error budget consumption"
              runbook: "https://wiki.internal/runbooks/slow-error-burn"
    ```
    
    ### PromQL Golden Signal Queries
    
    ```promql
    # Latency — 99th percentile request duration
    histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))
    
    # Traffic — requests per second by service
    sum(rate(http_requests_total[5m])) by (service)
    
    # Errors — error rate ratio
    sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
      /
    sum(rate(http_requests_total[5m])) by (service)
    
    # Saturation — CPU throttling ratio
    sum(rate(container_cpu_cfs_throttled_seconds_total[5m])) by (pod)
      /
    sum(rate(container_cpu_cfs_periods_total[5m])) by (pod)
    ```
    
    ### Toil Automation Script (Python)
    
    ```python
    #!/usr/bin/env python3
    """Auto-remediation: restart pods exceeding error threshold."""
    import subprocess, sys, json
    
    ERROR_THRESHOLD = 0.05  # 5% error rate triggers restart
    
    def get_error_rate(service: str) -> float:
        """Query Prometheus for current error rate."""
        import urllib.request
        query = f'sum(rate(http_requests_total{{status=~"5..",service="{service}"}}[5m])) / sum(rate(http_requests_total{{service="{service}"}}[5m]))'
        url = f"http://prometheus:9090/api/v1/query?query={urllib.request.quote(query)}"
        with urllib.request.urlopen(url) as resp:
            data = json.load(resp)
        results = data["data"]["result"]
        return float(results[0]["value"][1]) if results else 0.0
    
    def restart_deployment(namespace: str, deployment: str) -> None:
        subprocess.run(
            ["kubectl", "rollout", "restart", f"deployment/{deployment}", "-n", namespace],
            check=True
        )
        print(f"Restarted {namespace}/{deployment}")
    
    if __name__ == "__main__":
        service, namespace, deployment = sys.argv[1], sys.argv[2], sys.argv[3]
        rate = get_error_rate(service)
        print(f"Error rate for {service}: {rate:.2%}")
        if rate > ERROR_THRESHOLD:
            restart_deployment(namespace, deployment)
        else:
            print("Within SLO threshold — no action required")
    ```
    
    Maintained by [@jeffallan](https://github.com/jeffallan), Principal Consultant at [Synergetic Solutions](https://synergetic.solutions)
    
    [Documentation](https://jeffallan.github.io/claude-skills/skills/devops/sre-engineer/)
    

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.