Remote ML Evaluation Specialist - Shape AI Benchmarks

Alignerr Corp.; Toronto; Remote; CAD 83,000 - 165,000; Part time

Job summary

Alignerr is seeking domain experts with deep machine learning knowledge to design evaluation challenges that push state-of-the-art AI systems to their limits. This remote, hourly contract offers 10–40 hours per week and the chance to influence how AI models are measured, trained, and improved, by writing clear problem statements, evaluation criteria, and gold‑standard solutions.

Join a global team of researchers and engineers and help shape the future of AI evaluation research while enjoying

Qualifications

• Graduate-level expertise in ML intersecting with scientific/technical discipline.
• Strong working knowledge of ML methods: model selection, feature engineering, evaluation metrics, and pipeline design.
• Experience publishing or conducting original research is highly valued.
• Excellent written communication and ability to articulate complex problems clearly.
• Self-motivated and energized by independent work.

Responsibilities

• Design complex, original machine learning problems rooted in your area of domain expertise.
• Create evaluation tasks that demand advanced knowledge beyond standard ML pipelines.
• Draw from your research to craft challenges that test highly capable AI models.
• Write clear problem statements, define evaluation criteria, and establish gold-standard solutions.
• Assess AI-generated solutions for correctness, creativity, and methodological rigor.
• Document problem difficulty, domain knowledge, and expected failure modes.
• Collaborate asynchronously with a global team of researchers and engineers.

Skills

ML methods knowledge

Research experience

Communication skills

Independent work

Education

MS or PhD preferred

Job description

Alignerr is seeking domain experts with deep machine learning knowledge to design evaluation challenges that push state-of-the-art AI systems to their limits. This remote, hourly contract offers 10–40 hours per week and the chance to influence how AI models are measured, trained, and improved, by writing clear problem statements, evaluation criteria, and gold‑standard solutions.

Join a global team of researchers and engineers and help shape the future of AI evaluation research while enjoying