AI Systems Engineer: Debug & Build Agent Infrastructure

Neonhealth; San Francisco (CA); On-site; USD 140,000 - 210,000; Full time

Job summary

Neonhealth is seeking a Software Engineer to improve the real-world behavior of AI systems, build agent simulators, and design LLM evals and QA tooling. You’ll work hands-on with tracing runtime issues, debugging production systems, and interfacing with client data in a fast-paced healthcare automation environment.

You’ll tackle prompts, evaluation datasets, and tooling to push the boundaries of agent performance while maintaining reliability, observability, and scalable infrastructure in

Qualifications

• Proficiency in TypeScript and generalist software engineering.
• Strong debugging skills with ability to trace runtime failures.
• Experience with LLM systems, prompts, and evals is a plus.
• Ability to transform unstructured client data for pipelines.
• Proficient with Git and modern development workflows.

Responsibilities

• Trace and fix runtime bugs, then write regression tests.
• Design evaluation datasets to simulate realistic workflows or red-team systems.
• Build internal tooling for QA and agent simulation.
• Normalize and transform messy client data for system integration.
• Set up automatic testing and latency tracking infrastructure.
• Create dashboards and observability tooling for agentic system behavior.
• Expand on eval and testing frameworks and agent simulation infrastructure.

Skills

TypeScript

Debugging

LLM systems

Data transformation

Git

System design

Tools

Cursor

GitHub Copilot

Claude

Job description

Neonhealth is seeking a Software Engineer to improve the real-world behavior of AI systems, build agent simulators, and design LLM evals and QA tooling. You’ll work hands-on with tracing runtime issues, debugging production systems, and interfacing with client data in a fast-paced healthcare automation environment.

You’ll tackle prompts, evaluation datasets, and tooling to push the boundaries of agent performance while maintaining reliability, observability, and scalable infrastructure in