Lead ML Engineer - LLM/Agentic Evaluation Systems

Jobtailor; California (MO); On-site; USD 140,000 - 210,000; Full time

Job summary

Jobtailor is seeking an engineer to design and develop ML evaluation systems for agentic and LLM-based architectures. You will translate research ideas into production-grade ML systems with measurable impact and partner with Research, Product, and Platform teams to productize experiments into robust AI solutions.

Stay current with advancements in ML, NLP, and LLM systems and contribute to technical discussions.

Qualifications

• Deep experience in NLP and modern ML systems, including hands-on work with LLM-based or agentic systems.
• Strong architectural skills and proven experience designing complex software systems.
• Production experience with Python, AWS, Kubernetes, and/or Docker.
• Experience implementing technical solutions and tackling challenging engineering problems.
• Bachelor’s Degree in CS or another related field.
• Experience building and evaluating agentic systems at scale.
• Production experience with LLM-centric services, including inference, orchestration, evaluation, and monitoring.
• Familiarity with large-scale ML experimentation, benchmarking, or simulation frameworks.
• Knowledge of techniques for optimizing model architectures for faster inference.
• Experience with AWS, CI/CD, Kafka, and Athena.
• Proficiency in technical mentorship of junior and mid-level engineers.
• Willingness to learn, collaborate across teams, and teach others.

Responsibilities

• Design and develop ML evaluation systems for agentic and LLM-based architectures.
• Translate research ideas into production-grade ML systems with measurable impact.
• Partner with Research, Product, and Platform teams to productize experiments into robust AI solutions.
• Stay current with advancements in ML, NLP, and LLM systems and contribute to technical discussions.
• Mentor and support other engineers through design reviews, feedback, and knowledge sharing.
• Build and evaluate the core intelligence behind agentic AI systems.
• Design and own evaluation frameworks ensuring quality, safety, and performance across complex agentic systems.

Skills

NLP expertise

LLM-based systems

Architectural design

Python development

AWS

Kubernetes

Docker

CI/CD

Kafka

Athena

Education

Bachelor's Degree in CS

Tools

AWS

Kubernetes

Docker

CI/CD

Kafka

Athena

Job description

Jobtailor is seeking an engineer to design and develop ML evaluation systems for agentic and LLM-based architectures. You will translate research ideas into production-grade ML systems with measurable impact and partner with Research, Product, and Platform teams to productize experiments into robust AI solutions.

Stay current with advancements in ML, NLP, and LLM systems and contribute to technical discussions.