LLM Agent Evaluation Scientist

ByteDance; Seattle (WA); On-site; USD 180,000 - 240,000; Full time

Job summary

ByteDance in Seattle is seeking engineers and researchers to join the Applied Machine Learning Ark team. You will help build MaaS platforms for large language models and contribute to end-to-end MaaS solutions across markets.

You will design evaluation systems, run benchmarks, and analyze large-scale logs to drive model improvement, with opportunities to publish and pursue bold ideas within a leading tech company.

Qualifications

• PhD in Computer Science / AI / ML / Data Science required or recently completed.
• Solid foundation in machine learning and deep learning.
• Hands-on experience with LLM-based systems through research or projects.
• Strong Python skills and experience with an ML or agent evaluation framework.
• Demonstrated research or engineering ability through publications or internships.

Responsibilities

• Design evaluation systems for LLM-based agents, covering task success, tool use, reasoning quality, and reliability.
• Build benchmarks and automated judging pipelines, combining rule-based checks, model-based judging, and human review.
• Analyze agent execution traces and user feedback to identify failure patterns and turn them into concrete system improvements.
• Support the closed loop from experience to capability, and work with research, platform, and product teams to bring methods into production.

Skills

Python

ML fundamentals

LLM-based systems

Research experience

Publications

Education

PhD in Computer Science / AI / ML / Data Science

Tools

ML evaluation framework

Agent tooling

Job description

ByteDance in Seattle is seeking engineers and researchers to join the Applied Machine Learning Ark team. You will help build MaaS platforms for large language models and contribute to end-to-end MaaS solutions across markets.

You will design evaluation systems, run benchmarks, and analyze large-scale logs to drive model improvement, with opportunities to publish and pursue bold ideas within a leading tech company.