Evaluations Team Lead – AI Benchmark Platform (Remote)

Jobgether SRL; Portugal; Remote; EUR 60,000 - 100,000; Full time

Benefits offered by this job

Competitive compensation with equity

Health coverage for employees and衔

Job summary

Jobgether SRL, on behalf of a partner company, is seeking an Evaluations Team Lead in Portugal to shape how enterprise AI models are evaluated, validated, and improved in production. You will build a shared evaluation platform and define rigorous benchmarking, data quality, and reproducibility standards while staying hands-on with Python and experiments.

You will recruit and mentor a small team, collaborate with research, engineering, and customer-facing teams, and drive evidence-based

Qualifications

• Proven experience owning the evaluation of tabular machine learning systems used in production environments or high‑impact customer decision‑making.
• Strong statistical expertise, including selecting appropriate metrics and validation schemes, estimating uncertainty, comparing models across datasets, and accounting for repeated experimentation.
• Practical experience identifying and preventing data leakage involving preprocessing, feature engineering, joins, temporal dependencies, and related entities across training and evaluation splits.
• Strong proficiency in Python and SQL, with familiarity with scikit‑learn, gradient‑boosted decision trees, and the development of reliable ML tools or platforms used by multiple teams.
• Demonstrated experience designing fair and reproducible model comparisons, including hyperparameter optimization, resource allocation, and end‑to‑end inference latency measurement.
• Previous people management experience, including hiring, technical mentoring, team development, and performance management, combined with a willingness to remain actively involved in technical work.
• Excellent written and verbal communication skills, with the ability to collaborate effectively with researchers, software engineers, and customer data scientists.
• A strong analytical mindset and the confidence to challenge unsupported conclusions, communicate uncertainty, and ensure that technical decisions are grounded in reliable evidence.
• Experience evaluating tabular foundation models or automated machine learning (AutoML) systems is an advantage.
• Familiarity with measuring the impact of model optimizations on predictive performance and inference efficiency is desirable.
• Experience working with relational or multi‑table datasets and data platforms such as Snowflake or Databricks is a plus.
• Knowledge of automated or agent‑driven machine learning workflows, including identifying failure modes that aggregate metrics may conceal, is beneficial.

Responsibilities

• Build and maintain a shared evaluation platform that supports research, engineering, and Applied AI teams, enabling independent benchmarking and investigation of model performance.
• Develop and continuously improve benchmark suites, curated datasets, and model comparisons to assess predictive quality across a broad range of business use cases.
• Establish rigorous evaluation standards covering statistical metrics, validation strategies, data splits, leakage prevention, calibration, uncertainty estimation, and benchmark contamination.
• Design reproducible evaluation pipelines with version-controlled inputs, datasets, and artifacts, integrating automated regression checks into research and release workflows.
• Benchmark models against competing approaches using fair and consistent protocols for hyperparameter tuning, data access, computational resources, and end-to-end latency measurement.
• Evaluate predictive performance, inference latency, and cost efficiency across deployment configurations, task types, and dataset characteristics.
• Support customer proof‑of‑concept evaluations by providing reliable tooling, methodological guidance, and data‑driven analysis to help demonstrate model value.
• Translate evaluation findings into actionable research priorities, release recommendations, and evidence‑based improvement plans for customer use cases.
• Recruit, lead, and develop a small team of evaluation specialists, setting priorities, providing technical coaching, and delivering constructive performance feedback.
• Remain hands‑on with Python development, experimental design, statistical analysis, and evaluation methodology while fostering collaboration across research, engineering, and customer‑facing teams.

Job description

Jobgether SRL, on behalf of a partner company, is seeking an Evaluations Team Lead in Portugal to shape how enterprise AI models are evaluated, validated, and improved in production. You will build a shared evaluation platform and define rigorous benchmarking, data quality, and reproducibility standards while staying hands-on with Python and experiments.

You will recruit and mentor a small team, collaborate with research, engineering, and customer-facing teams, and drive evidence-based