LLM MaaS Engineer – Agents & Evaluation

ByteDance; San Jose (CA); On-site; USD 128,000 - 256,000; Full time

Job summary

ByteDance in California's San Jose area seeks engineers and researchers for the Applied Machine Learning Ark team to build MaaS platforms and end-to-end LLM services, spanning training, inference, and multi-model workflows.

The role emphasizes designing evaluation systems for LLM-based agents, building benchmarks, and turning failure patterns into concrete improvements that influence product decisions at scale.

Qualifications

• Bachelor’s/Master’s in Computer Science, AI, ML, Data Science or related field.
• Solid foundation in machine learning and deep learning.
• Hands-on experience with LLM-based systems (agents, tool calling, retrieval, multi-agent systems).
• Strong Python skills and experience with a mainstream ML or agent evaluation framework.
• Publications or projects demonstrating research or engineering ability.

Responsibilities

• Design evaluation systems for LLM-based agents covering task success, tool use, reasoning quality, and reliability.
• Build benchmarks and automated judging pipelines using rule-based checks, model-based judging, and human review.
• Analyze agent traces and user feedback to identify failure patterns and drive system improvements.
• Collaborate with research, platform, and product teams to bring methods into production.

Skills

Python

LLM-based systems

Research

Machine learning

Tool calling/retrieval

Education

Bachelor's/Master's in CS/AI/ML/Data Science

Tools

ML frameworks

Agent evaluation framework

Job description

ByteDance in California's San Jose area seeks engineers and researchers for the Applied Machine Learning Ark team to build MaaS platforms and end-to-end LLM services, spanning training, inference, and multi-model workflows.

The role emphasizes designing evaluation systems for LLM-based agents, building benchmarks, and turning failure patterns into concrete improvements that influence product decisions at scale.