LLM Agent Evaluation Engineer — Build Intelligent Agents

Bytedance; San Jose (CA); On-site; USD 154,000 - 256,000; Full time

Job summary

ByteDance Applied Machine Learning Ark team in San Jose develops and operates LLM service platforms that offer MaaS solutions to businesses and researchers, spanning model serving, tool calling, and analytics.

This full-time role emphasizes designing evaluation systems for LLM agents, building benchmarks and pipelines, analyzing traces and feedback, and collaborating with research, platform, and product teams to bring methods into production.

Qualifications

• Bachelor's or Master's in Computer Science, AI, ML, Data Science or related field.
• Solid foundation in machine learning and deep learning.
• Hands-on experience with LLM-based systems (agents, tool calling, retrieval, multi-agent systems) through research, internships, or projects.
• Strong Python skills and experience with a mainstream ML or agent evaluation framework.
• Publications or open-source work demonstrating research or engineering ability.

Responsibilities

• Design evaluation systems for LLM-based agents, covering task success, tool use, reasoning quality, and reliability.
• Build benchmarks and automated judging pipelines, combining rule-based checks, model-based judging, and human review.
• Analyze agent execution traces and user feedback to identify failure patterns and turn them into concrete system improvements.
• Support the closed loop from experience to capability, and work with research, platform, and product teams to bring methods into production.

Skills

Python

ML fundamentals

LLM-based systems

Research/publications

Education

Bachelor's/Master's degree in CS/AI/ML/Data Science

Tools

Evaluation framework

Job description

ByteDance Applied Machine Learning Ark team in San Jose develops and operates LLM service platforms that offer MaaS solutions to businesses and researchers, spanning model serving, tool calling, and analytics.

This full-time role emphasizes designing evaluation systems for LLM agents, building benchmarks and pipelines, analyzing traces and feedback, and collaborating with research, platform, and product teams to bring methods into production.