ML Engineer — Evaluation & AI Agent Quality

Coda; Porto; Híbrido; EUR 90.000 - 120.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Hybrid work model

Home office setup support

Wellness stipends

Learning and development opportunities

Relocation support to Berlin

Descripción de la vacante

Superhuman is hiring a foundational Machine Learning Engineer to join our Porto team and build the evaluation foundation behind the next generation of AI-native productivity on our agent platform. You will own how we define, measure, and improve quality, building eval datasets, the infrastructure around them, and the metrics that tell us whether our models and agents are actually getting better for users.

This is a high-autonomy, high-impact role on a foundational team that moves fast from idea

Formación

• 5+ years of industry experience as a machine learning engineer or applied research scientist.
• Strong foundation in classic ML and NLP, including experimental design, statistics, and metric design, plus working knowledge of modern LLM approaches.
• Hands-on experience building evaluation datasets and the tooling and infrastructure to run evals at scale (data pipelines, labeling workflows, benchmarking frameworks).
• Experience using evaluation results to measure and improve model or product quality, whether for ranking, classification, or LLM-based systems.
• Strong Python and data engineering skills, with experience in reproducible, production-grade ML pipelines.
• Deep care about the end-user experience and a drive to ensure high quality.
• Ability to work independently with minimal guidance, manage priorities across multiple projects, and collaborate effectively with cross-functional teams in a fast-paced, results-driven environment.
• Nice to have: experience with LLM-as-judge, agent or workflow evaluation, or search/ranking systems; MS/PhD in Computer Science or a related field.

Responsabilidades

• Build high-quality evaluation datasets: Design, source, and curate golden sets, labeled data, and synthetic data for agent behaviors, search ranking, and proactive suggestions.
• Build the eval infrastructure: Create scalable, reproducible pipelines for running offline evaluations, regression testing, and benchmarking across models, prompts, and agent configurations.
• Measure and improve quality: Define metrics that reflect real user value, from classic ML metrics to LLM-as-judge and human-eval approaches.
• Own classic ML systems: Develop and deploy ML models for core product experiences, including search ranking and proactive suggestions that anticipate user needs.
• Support the agent platform: Partner with the team building orchestration, routing, and planning to define success criteria and quality gates for multi-agent workflows.
• Close the loop with production: Connect offline evals to online experiments and real-world signals, so we catch regressions early and learn from user experiences.
• Ship fast: Move quickly from idea to impact, with no red tape or endless approvals, on a platform used by millions.

Conocimientos

ML engineering

NLP fundamentals

Experimental design

Statistics

Metric design

Python

Data engineering

Independent work

Educación

MS/PhD in Computer Science or related field

Herramientas

LLM evaluation

Descripción del empleo

Superhuman is hiring a foundational Machine Learning Engineer to join our Porto team and build the evaluation foundation behind the next generation of AI-native productivity on our agent platform. You will own how we define, measure, and improve quality, building eval datasets, the infrastructure around them, and the metrics that tell us whether our models and agents are actually getting better for users.

This is a high-autonomy, high-impact role on a foundational team that moves fast from idea