Applied AI Engineer: Agent Evaluation & Benchmarks

micro1; United States; Remote; USD 200,000 - 250,000; Full time

Benefits offered by this job

Equity compensation

Bonuses

Health-insurance premium reimbursement

Paid time off

401(k) plan with company match

Remote-first workforce

Job summary

micro1 is seeking a Member of Technical Staff focused on applying AI research to real-world enterprise workflows. You will develop, evaluate, and improve LLM-powered systems, working with researchers, engineers, and customers to identify capability gaps and drive measurable improvements in performance.

The role emphasizes hands-on experimentation, telemetry-driven analysis, and building lightweight tooling to accelerate research and deployment in production environments.

Qualifications

• Experience in applied AI research and engineering.
• Ability to design evaluations for AI agents and complex workflows.
• Strong Python programming and building evaluation pipelines.
• Experience creating benchmarks, datasets, and metrics for AI systems.

Responsibilities

• Design and implement evaluation frameworks, benchmarks, and datasets for LLM-powered agents and complex AI workflows.
• Identify and analyze model, agent, and workflow failures in real-world production environments.
• Run rapid experiments to validate hypotheses and improve agent reliability.
• Collaborate with customers and researchers to evaluate, deploy, and optimize AI systems.
• Develop training and evaluation data, task specs, and feedback mechanisms to improve performance.
• Build Python-based evaluation pipelines and lightweight research tooling.
• Define evaluation strategies and optimization roadmaps with stakeholders.
• Contribute to reports, benchmarks, and research insights.

Skills

Applied AI research

LLM-based systems

Python

Evaluation pipelines

Tools

Research tooling

ML infrastructure

Job description

micro1 is seeking a Member of Technical Staff focused on applying AI research to real-world enterprise workflows. You will develop, evaluate, and improve LLM-powered systems, working with researchers, engineers, and customers to identify capability gaps and drive measurable improvements in performance.

The role emphasizes hands-on experimentation, telemetry-driven analysis, and building lightweight tooling to accelerate research and deployment in production environments.