Remote Applied AI Staff Engineer — Agent Evaluation

Zekai; United States; Remote; USD 200,000 - 250,000; Full time

Benefits offered by this job

Equity compensation

Performance-based bonuses

Health insurance reimbursement

401(k) plan with company match

Remote-first workforce

Job summary

Cortex Lab is seeking a hands-on Member of Technical Staff to develop, evaluate, and improve AI agents and LLM-powered systems operating in real-world enterprise workflows.

This role focuses on agent evaluations, failure analysis, and rapid experimentation, collaborating with researchers, engineers, domain experts, and enterprise customers to drive measurable improvements in AI system performance. Remote-first environment with strong compensation and benefits.

Qualifications

• Experience designing evaluations for AI agents.
• Ability to translate real-world problems into structured experiments.
• Strong Python and ML infrastructure skills.
• Experience working across research, engineering, and customers.

Responsibilities

• Design evaluation frameworks, benchmarks, and datasets for LLM-powered agents and complex AI workflows.
• Identify, analyze, and prioritize model, agent, and workflow failures in real-world production environments.
• Run rapid experiments to validate hypotheses, measure performance, and improve agent reliability and effectiveness.
• Partner with enterprise customers and researchers to evaluate, deploy, and optimize AI systems across production workflows.
• Develop high-quality training and evaluation data, task specifications, and feedback mechanisms to improve system performance.
• Build Python-based evaluation pipelines, experimentation tools, and lightweight research infrastructure.

Skills

Applied AI research

Python programming

Evaluation pipelines

LLM-based systems

Experimental tooling

Collaboration

Tools

ML infra tooling

Benchmarks & datasets

Python libraries

Job description

Cortex Lab is seeking a hands-on Member of Technical Staff to develop, evaluate, and improve AI agents and LLM-powered systems operating in real-world enterprise workflows.

This role focuses on agent evaluations, failure analysis, and rapid experimentation, collaborating with researchers, engineers, domain experts, and enterprise customers to drive measurable improvements in AI system performance. Remote-first environment with strong compensation and benefits.