Machine Learning Engineer – Evals

Mango Global Technologies; New York (NY); Sur place; USD 220 000 - 300 000; Plein temps

Avantages offerts par ce poste

Employee assistance program

Résumé du poste

Mango Global Technologies is seeking a Machine Learning Engineer for Evals to own end-to-end evaluation of an AI memory and identity system, shaping quality metrics and building evaluation infrastructure for rapid iteration across models.

The role requires 3+ years of LLM evaluation experience, production-grade Python, onsite NYC five days per week, and unrestricted US work authorization; visa sponsorship is not available.

Qualifications

• 3+ years of professional experience designing and owning LLM evaluations
• Experience designing, building, and running LLM evaluations end to end
• Production-quality Python experience including testing, maintainability, debugging and releases
• Experience creating evaluation methodologies, not just executing benchmarks
• Strong ability to investigate model or agent failures using traces, outputs and experiments
• Willingness to work onsite in New York City five days per week
• Unrestricted U.S. work authorization; visa sponsorship is not available

Responsabilités

• Define evaluation criteria to determine improvements in evolving representations
• Build infrastructure to measure improvements and track outcomes over time
• Use results to identify failures and drive fixes in engineering or research
• Design novel evaluation methodologies when ground truth is unavailable
• Develop scoring frameworks, metrics, rubrics, and diagnostic methodologies
• Create production-quality Python pipelines and evaluation harnesses
• Manage datasets, labels, experiment and model versions, and regression testing
• Integrate experiment tracking and observability into evaluation workflows
• Run experiments, document findings, and drive reruns and recommendations
• Own the evaluation process end-to-end from research question to recommendation

Connaissances

LLM evaluations

Python engineering

Trace analysis

Research mindset

Production code

Formation

CS/Math/ML degree

Outils

Python

PyTorch

Hugging Face/Transformers

Weights & Biases

Hydra

SQL

Kafka

Description du poste

The Machine Learning Engineer for Evals will own end-to-end evaluation for an AI memory and identity system, shaping how quality is measured and ensuring evaluation infrastructure supports rapid iteration across models, prompts, and representations.

Responsibilities

• Define how to determine whether evolving entity representations are improving.
• Build infrastructure needed to measure improvements and track evaluation outcomes over time.
• Use evaluation results to identify failures and drive fixes through engineering or research changes.
• Define what “better” means for representations that evolve across updates.
• Design novel evaluation methodologies when traditional ground truth is unavailable or insufficient.
• Develop scoring frameworks, metrics, rubrics, judge models, and diagnostic methodologies.
• Continuously evolve evaluation definitions as models and product capabilities change.
• Build production-quality Python pipelines and evaluation harnesses.
• Manage datasets, labels, experiment versions, model versions, judges, reruns, and regression testing.
• Create systems that help researchers and engineers answer new evaluation questions quickly.
• Integrate experiment tracking and observability into evaluation workflows.
• Analyze model outputs, agent trajectories, and execution traces.
• Identify meaningful failure modes instead of optimizing for superficial metrics.
• Run experiments, compare results, document findings, and execute the resulting rerun and recommendations.
• Develop and operate simulation environments capable of modeling large numbers of entities.
• Increase throughput and reliability of evaluation runs.
• Build systems that enable rapid iteration across models, prompts, representations, and evaluation strategies.
• Own the evaluation process end to end: research question → evaluation design → data → implementation → execution → analysis → diagnosis → rerun → recommendation.

Required Qualifications

• 3+ years of professional experience designing and owning LLM evaluations.
• Demonstrated experience designing, building, and running LLM evaluations end to end.
• Production-quality Python experience, including testing, maintainability, debugging, and release practices.
• Experience creating evaluation methodologies, not only executing existing benchmarks.
• Strong ability to investigate model or agent failures using traces, outputs, datasets, and experimental results.
• Ability to work onsite in New York City five days per week.
• Unrestricted U.S. work authorization; visa sponsorship is not available.

Technologies

• Python
• PyTorch
• Hugging Face / Transformers
• Weights & Biases
• Hydra
• SQL
• Kafka

Preferred Technical Skills

• Weights & Biases
• Hydra
• SQL
• Kafka or other streaming or event-driven infrastructure
• Evaluation and observability frameworks
• Agent simulation and multi-agent systems
• Large-scale data pipelines

Nice to Have

• Experience with AI memory, identity, personalization, agent evaluation, or agentic optimization.
• Research experience in LLMs, NLP, machine learning, evaluation, or related areas.
• First-author or co-author research at NeurIPS, ICML, ICLR, ACL, EMNLP, or comparable venues.
• Alternatively, a high-quality open-source project, benchmark, framework, or technical publication demonstrating comparable research ability.
• Degree in Computer Science, Mathematics, Statistics, Machine Learning, or another quantitative discipline.

Benefits

• Employee assistance program

Compensation

USD 220,000 - 300,000 base salary per year plus competitive equity. Final compensation depends on experience, technical depth, research contributions, and interview performance.

Hiring Process

The selected candidate should expect technical discussions focused on:

• Designing an evaluation from scratch.
• Diagnosing an LLM or agent failure from traces.
• Building a production evaluation pipeline.
• Research methodology and experimental design.
• Python and ML engineering.
• Previous research or open-source work.

Application Questions

• Can you work full-time on-site in New York City 5 days per week, and do you currently have unrestricted U.S. work authorization without requiring visa sponsorship?
• Have you either (a) published ML/LLM research at a recognized conference or journal or (b) released substantial open-source research or ML work of comparable quality?
• Do you have hands-on experience with PyTorch AND Hugging Face or Transformers?
• Do you have production-level Python experience, including writing tests, maintaining production code, debugging, and managing releases?
• How many years of professional experience do you have designing and owning LLM evaluations?
• Have you personally designed, built, and run an LLM evaluation system end to end, including defining the evaluation methodology or metrics rather than only executing an existing benchmark?
• Share your LinkedIn URL.

Work Location and Salary Details

• Location: New York, NY (onsite)
• Salary: USD 220,000 - 300,000 per year
• Minimum experience: 3 years

Not a Strong Fit

• Candidates who have only run established benchmarks without designing their own evaluations.
• Engineers whose LLM experience is limited to application development without evaluation ownership.
• Candidates without substantial hands-on Python engineering experience.
• Candidates without evidence of research thinking, novel experimentation, published work, or comparable open-source work.
• Candidates unwilling or unable to work five days per week from the New York City office.