LLM Agent Evaluation & Evolution Architect

ByteDance; San Jose (CA); On-site; USD 162,000 - 317,000; Full time

Benefits offered by this job

Medical, dental, and vision insurance

401(k) with company match

Paid parental leave

Short-/long-term disability coverage

Life insurance

Wellbeing benefits

Paid holidays

Job summary

ByteDance's Applied Machine Learning Ark team develops end-to-end MaaS platforms that combine system engineering with large language models to serve businesses globally. The US team designs and operates MaaS solutions across the US and international markets beyond mainland China.

We seek PhD-level researchers and engineers with a solid ML background, Python skills, and hands-on LLM experience to build evaluation pipelines, benchmarks, and production-grade components for scalable AI systems.

Qualifications

• PhD in CS/AI/ML/Data Science or related field is required or recently completed.
• Solid foundation in machine learning and deep learning.
• Hands-on experience with LLM-based systems (agents, tool calls, retrieval, multi-agent systems).
• Strong Python skills and experience with a mainstream ML or agent evaluation framework.
• Demonstrated research or engineering ability through publications, projects, internships, or open-source work.

Responsibilities

• Design evaluation systems for LLM-based agents, covering task success, tool use, reasoning quality, and reliability.
• Build benchmarks and automated judging pipelines with rule-based checks, model-based judging, and human review.
• Analyze agent execution traces and user feedback to identify failure patterns and drive improvements.
• Collaborate with research, platform, and product teams to bring methods into production.

Skills

Solid ML foundation

Hands-on with LLM-based systems

Strong Python skills

Publications/open-source work

Demonstrated research/engineering

Education

PhD in Computer Science/AI/ML/Data Science

Job description

ByteDance's Applied Machine Learning Ark team develops end-to-end MaaS platforms that combine system engineering with large language models to serve businesses globally. The US team designs and operates MaaS solutions across the US and international markets beyond mainland China.

We seek PhD-level researchers and engineers with a solid ML background, Python skills, and hands-on LLM experience to build evaluation pipelines, benchmarks, and production-grade components for scalable AI systems.