LLM Systems Engineer: AI Agents & Evaluation

ByteDance; Seattle (WA); On-site; USD 120,000 - 180,000; Full time

Job summary

ByteDance is hiring in Seattle for roles within the Applied Machine Learning Ark team. You will help design, develop, and operate MaaS platforms using LLMs and multimodal models, spanning training, inference, prompt engineering, and evaluation pipelines.

The team builds end-to-end solutions, analyzes large-scale log data, and collaborates across research, platform, and product groups to drive product improvements. A strong background in ML and Python is essential.

Qualifications

• Bachelor's or Master's in CS/AI/ML/DS or related field.
• Solid foundation in machine learning and deep learning.
• Hands-on experience with LLM-based systems (agents, tool calling, retrieval, multi-agent systems).
• Strong Python skills and experience with a mainstream ML or agent evaluation framework.
• Publications, substantial projects, internships, or open-source work.

Responsibilities

• Design evaluation systems for LLM-based agents, covering task success, tool use, reasoning quality, and reliability.
• Build benchmarks and automated judging pipelines, combining rule-based checks, model-based judging, and human review.
• Analyze agent execution traces and user feedback to identify failure patterns and turn them into concrete system improvements.
• Support the closed loop from experience to capability, and work with research, platform, and product teams to bring methods into production.

Skills

ML fundamentals

Python

LLM-based systems

Research or projects

Publication or open-source work

Education

Bachelor's or Master's in CS/AI/ML/DS

Tools

ML evaluation framework

Agent evaluation pipelines

Job description

ByteDance is hiring in Seattle for roles within the Applied Machine Learning Ark team. You will help design, develop, and operate MaaS platforms using LLMs and multimodal models, spanning training, inference, prompt engineering, and evaluation pipelines.

The team builds end-to-end solutions, analyzes large-scale log data, and collaborates across research, platform, and product groups to drive product improvements. A strong background in ML and Python is essential.