ML Engineer - Evaluation & AI Agent Platform

Superhuman; Porto; Híbrido; EUR 70 000 - 90 000; Tempo integral

Vantagens oferecidas por esta oferta de emprego

Relocation to Berlin

Home office setup

Wellness stipend

Learning & development

Resumo da oferta

Superhuman seeks a foundational Machine Learning Engineer to own evaluation datasets and the metrics that determine model quality for its agent platform. You will build scalable offline evaluation pipelines and drive improvements in ranking, proactive suggestions, and end-user experiences.

The role emphasizes high autonomy, fast iteration, and collaboration with cross-functional teams. A hybrid working model in Porto enables focused work with in-person collaboration, and there is relocation

Qualificações

• 5+ years of industry experience as a machine learning engineer or applied research scientist.
• Strong foundation in classic ML and NLP, incl. experimental design, statistics, and metric design, plus working knowledge of modern LLM approaches.
• Hands-on experience building evaluation datasets and the tooling and infrastructure to run evals at scale (data pipelines, labeling workflows, benchmarking frameworks).
• Experience using evaluation results to measure and improve model or product quality, whether for ranking, classification, or LLM-based systems.
• Strong Python and data engineering skills, with experience in reproducible, production-grade ML pipelines.
• Deep care about the end-user experience and a drive to ensure high quality.
• Ability to work independently with minimal guidance, manage priorities across multiple projects, and collaborate effectively with cross-functional teams in a fast-paced, results-driven environment.
• Nice to have: MS/PhD in CS or related field.
• Nice to have: experience with LLM-as-judge, agent or workflow evaluation, or search/ranking systems.

Responsabilidades

• Build high-quality evaluation datasets, including golden sets, labeled and synthetic data for agent behaviors and ranking.
• Build scalable eval infrastructure for offline evaluations, regression testing, and benchmarking across models and configurations.
• Define metrics reflecting real user value and validate their correlation with user outcomes.
• Develop and deploy ML models for core product experiences, including search ranking and proactive suggestions.
• Support the agent platform by defining success criteria and quality gates for multi-agent workflows.
• Connect offline evals to online experiments and real-world signals to catch regressions.
• Move quickly from idea to impact with a fast, autonomous workflow.

Conhecimentos

Python

Data engineering

ML pipelines

NLP fundamentals

Experimental design

LLM approaches

Independent work

Formação académica

MS/PhD in CS

Ferramentas

Pandas

SQL

ML frameworks

Descrição da oferta de emprego

Superhuman seeks a foundational Machine Learning Engineer to own evaluation datasets and the metrics that determine model quality for its agent platform. You will build scalable offline evaluation pipelines and drive improvements in ranking, proactive suggestions, and end-user experiences.

The role emphasizes high autonomy, fast iteration, and collaboration with cross-functional teams. A hybrid working model in Porto enables focused work with in-person collaboration, and there is relocation