LLM Evaluation Architect: End-to-End Pipelines

Mango Global Technologies; New York (NY); On-site; USD 220,000 - 300,000; Full time

Benefits offered by this job

Employee assistance program

Job summary

Mango Global Technologies is seeking a Machine Learning Engineer for Evals to own end-to-end evaluation of an AI memory and identity system, shaping quality metrics and building evaluation infrastructure for rapid iteration across models.

The role requires 3+ years of LLM evaluation experience, production-grade Python, onsite NYC five days per week, and unrestricted US work authorization; visa sponsorship is not available.

Qualifications

• 3+ years of professional experience designing and owning LLM evaluations
• Experience designing, building, and running LLM evaluations end to end
• Production-quality Python experience including testing, maintainability, debugging and releases
• Experience creating evaluation methodologies, not just executing benchmarks
• Strong ability to investigate model or agent failures using traces, outputs and experiments
• Willingness to work onsite in New York City five days per week
• Unrestricted U.S. work authorization; visa sponsorship is not available

Responsibilities

• Define evaluation criteria to determine improvements in evolving representations
• Build infrastructure to measure improvements and track outcomes over time
• Use results to identify failures and drive fixes in engineering or research
• Design novel evaluation methodologies when ground truth is unavailable
• Develop scoring frameworks, metrics, rubrics, and diagnostic methodologies
• Create production-quality Python pipelines and evaluation harnesses
• Manage datasets, labels, experiment and model versions, and regression testing
• Integrate experiment tracking and observability into evaluation workflows
• Run experiments, document findings, and drive reruns and recommendations
• Own the evaluation process end-to-end from research question to recommendation

Skills

LLM evaluations

Python engineering

Trace analysis

Research mindset

Production code

Education

CS/Math/ML degree

Tools

Python

PyTorch

Hugging Face/Transformers

Weights & Biases

Hydra

SQL

Kafka

Job description

Mango Global Technologies is seeking a Machine Learning Engineer for Evals to own end-to-end evaluation of an AI memory and identity system, shaping quality metrics and building evaluation infrastructure for rapid iteration across models.

The role requires 3+ years of LLM evaluation experience, production-grade Python, onsite NYC five days per week, and unrestricted US work authorization; visa sponsorship is not available.