Expert: Interactive Coding, Site Reliability Engineering (Contract, Remote)

Patronus AI, Inc.; Oregon; Hybrid; USD 170,000 - 243,000; Full time

Benefits offered by this job

Paid trial session

Remote work

Job summary

Patronus AI, Inc. is recruiting a production systems engineer for live, on-call style reliability sessions. You will work with engineers on a sandboxed stack (Kubernetes, Prometheus, Grafana, Helm, Terraform) to assess scenarios, fix issues, and document resolutions.

Strong focus on reasoning, correctness, and clear communication in English. Ideal candidates have 3+ years of production experience, hands-on with the listed tools, and a track record of analyzing dashboards, logs, and incidents.

Qualifications

• 3+ years running production systems with on-call experience in the last 2 years.
• Hands-on with Kubernetes, Prometheus and Grafana, or comparable tools.
• Familiar with a log stack (Loki/ELK/Splunk) and infrastructure as code (Terraform/Helm).
• Proficient in Python, Go or Bash and reading unfamiliar codebases.
• Ability to explain reasoning aloud in professional English.
• Reliable for live sessions and able to provide evidence for conclusions.

Responsibilities

• Work live with a research team on production-style reliability scenarios.
• Read dashboards/logs, write PromQL and LogQL, fix Helm/Kubernetes configs.
• Roll back bad deploys and write post-mortems.
• Explain fixes considered and rejected as you work.
• Review scenarios for realism, completeness, and gaps.
• Write notes after sessions on approach and verifications.

Skills

Kubernetes

Prometheus & Grafana

Logging stack (Loki/ELK/Splunk)

Terraform & Helm

CI/CD

Python / Go / Bash

English communication

Job description

About Patronus AIPatronus AI is a frontier lab developing simulation research and infrastructure to accelerate progress toward human-aligned AGI. We are on a mission to simulate all of the world’s intelligence.We are the team behind some of the earliest and most influential research in AI evaluation like FinanceBench , Lynx, SimpleSafetyTests , CopyrightCatcher , Humanity's Last Exam , and more. We are formerly AI researchers and engineers from companies like Meta AI, Amazon AGI, and Google. Our customers include foundation model labs and Fortune 500 enterprises like Adobe. We are backed by top-tier investors like Lightspeed Venture Partners, Notable Capital, Stanford University, Noam Brown, Gokul Rajaram, and more.

At a glance

• Engagement: Independent contractor, fully remote. Live screen-share sessions with our researchers plus short asynchronous write-ups.
• Pay: $150 per hour. The trial session is paid.
• Commitment: About 10 hours per week, 1 to 3 months to start, with extension for strong performers.
• Start: Rolling
• Location: Remote, any country, with a few hours of overlap with US or EU working hours.

The role

You’ll work live with our research team on production-style reliability scenarios in a sandboxed stack: Kubernetes, Prometheus and Grafana, a log system, Helm and Terraform, and CI/CD. You solve the scenario the way you would on call, explain your reasoning as you go, tell us where the scenario doesn’t match real operations, and help define what a correct resolution looks like. Each session starts from a scenario we provide. You investigate it on a shared screen, fix or mitigate it, and then write a short note on how you approached it. Your work becomes evaluation and training data for frontier models. You don’t need an AI background. You do need to be the kind of engineer who reads the dashboard before touching the config, and who can say out loud why a tempting fix is the wrong one.

Areas we’re hiring for

• Incident response and on-call. Engineers on a production rotation who triage alerts, read metrics, logs and traces, find root cause, mitigate, roll back and write the post-mortem. Typical titles: Site Reliability Engineer, Production Engineer, On-Call Engineer, Platform Engineer.
• Platform and infrastructure. Engineers who own clusters, deploy pipelines and infrastructure as code: Kubernetes, Helm, Terraform, CI/CD, networking and capacity. Typical titles: Infrastructure Engineer, DevOps Engineer, Platform Engineer, Cloud Engineer.
• Observability. Engineers who build and tune the monitoring itself: Prometheus and PromQL, Grafana dashboards and alert rules, Loki or ELK, OpenTelemetry tracing, SLOs and error budgets. Typical titles: Observability Engineer, Monitoring Engineer, Site Reliability Engineer (Observability).

What you’ll do

• Work through incident, debugging and infrastructure-change scenarios live with a researcher: reading dashboards and logs, writing PromQL and LogQL, fixing Helm and Kubernetes configuration, rolling back bad deploys and writing post-mortems.
• Explain your reasoning as you work, including the fixes you considered and rejected.
• Review scenarios we have drafted and say plainly what is unrealistic, too easy or underspecified.
• Help write the acceptance criteria for each scenario, including the tempting wrong moves an engineer under pressure might make.
• Write short notes after each session on how you approached the problem and what you verified.
• Work in reviewer feedback within agreed turnaround times, and flag guideline gaps or edge cases you run into.

Who we’re looking for

Required

• 3+ years running production systems in at least one of the areas above, including time on an on-call rotation within the last two years. You did the work yourself; managing the team is not enough.
• Hands‑on with Kubernetes, Prometheus and Grafana (or Datadog or New Relic), a log stack such as Loki, ELK or Splunk, infrastructure as code such as Terraform and Helm, and a CI/CD system.
• Comfortable in Python, Go or Bash and at reading unfamiliar codebases.
• Able to explain your reasoning out loud while you work, in clear professional English.
• Reliable availability for scheduled live sessions and the habit of showing the evidence behind every conclusion.

Nice to have

• Experience writing runbooks, post-mortems, SLO definitions or incident training material.
• Familiarity with how AI coding agents fail on operations work.

Our standards

• All solutions, write-ups and reviews are your own work. LLM-generated code or text is not permitted.
• Scenarios and solutions must be original. Nothing may be copied from published or easily searchable sources.
• Never include code, configuration, runbooks or incident details from a current or former employer or client, or anything covered by an NDA. All systems and scenarios are fictional, and all work is your own.
• Quality is judged on correctness, clarity of reasoning and reviewer agreement, not on speed or volume.

How hiring works

• Short AI interview and resume review
• One paid trial session, scored against a fixed quality checklist
• Onboarding and your first project

Patronus AI is an equal opportunity employer. We celebrate diversity in our workplace, and all qualified applicants will receive consideration for employment without regard to age, ancestry, color, family or medical care leave, gender identity or expression, genetic information, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran status, race, religion, sex (including pregnancy), sexual orientation, or other legally protected characteristics.