Remote Senior AI Engineer: GenAI Evaluation Frameworks

Grafana Labs; Canada; Remote; CAD 153,000 - 185,000; Full time

Job summary

A leading observability tools company is seeking a Senior Engineer specializing in Generative AI and ML Evaluation Frameworks. This remote position focuses on designing evaluation frameworks and scaling automated pipelines for AI systems. Ideal candidates should have experience in Generative AI evaluation, collaborative problem-solving skills, and a passion for building efficient AI-supported systems. A competitive compensation range is offered for Canadian applicants.

Benefits

Equity

Bonus

Comprehensive benefits package

Qualifications

• Experience designing and implementing evaluation frameworks for AI/ML systems.
• High autonomy to collaborate and translate team goals into clear, testable criteria.

Responsibilities

• Design and implement robust evaluation frameworks for GenAI and LLM-based systems.
• Develop tooling for automated evaluation of model outputs.
• Lead development of dataset management processes for GenAI evaluation.

Skills

Evaluating Generative AI systems

Prompt engineering

Structured output evaluation

Context-window management in LLM systems

Collaborative problem-solving

Job description

A leading observability tools company is seeking a Senior Engineer specializing in Generative AI and ML Evaluation Frameworks. This remote position focuses on designing evaluation frameworks and scaling automated pipelines for AI systems. Ideal candidates should have experience in Generative AI evaluation, collaborative problem-solving skills, and a passion for building efficient AI-supported systems. A competitive compensation range is offered for Canadian applicants.