Senior Software Engineer, GenAI & ML Evaluation Frameworks - Grafana Ops, AI/ML (Remote, Canada)

Grafana Labs; Canada; Remote; CAD 153,000 - 185,000; Full time

Job summary

Grafana Labs is seeking a Senior Software Engineer to evaluate Generative AI and LLM systems, contributing to the development of internal evaluation frameworks and automated pipelines. This remote position requires expertise in AI/ML evaluation, suited for a candidate who thrives in fast-paced, innovative environments.

Qualifications

• Experience designing and implementing evaluation frameworks for AI/ML systems.
• Familiarity with prompt engineering and LLM systems.
• High autonomy in collaborating on team goals.

Responsibilities

• Design and implement evaluation frameworks for GenAI and LLM-based systems.
• Develop tooling for automated evaluation of model outputs.
• Lead dataset management processes for GenAI evaluation.

Skills

Designing evaluation frameworks

Prompt engineering

Context-window management

Job description

Senior Software Engineer, GenAI & ML Evaluation Frameworks - Grafana Ops, AI/ML (Remote, Canada)

This is a remote opportunity and we would be interested in applicants from Canada time zones only at this time.

At Grafana, we build observability tools that help users understand, respond to, and improve their systems – regardless of scale, complexity, or tech stack. The Grafana AI teams play a key role in this mission by helping users make sense of complex observability data through AI-driven features. These capabilities reduce toil, lower the barrier of domain expertise, and surface meaningful signals from noisy environments.

We are looking for an experienced engineer with expertise in evaluating Generative AI systems, particularly Large Language Models (LLMs), to help us build and evolve our internal evaluation frameworks, and/or integrate existing best-of-breed tools. This role involves designing and scaling automated evaluation pipelines, integrating them into CI/CD workflows, and defining metrics that reflect both product goals and model behavior. As the team matures, there’s a broad opportunity to expand or redefine this role based on impact and initiative.

The kind of problems you’ll be tackling:

• Design and implement robust evaluation frameworks for GenAI and LLM-based systems, including golden test sets, regression tracking, LLM-as-judge methods, and structured output verification.
• Develop tooling to enable automated, low-friction evaluation of model outputs, prompts, and agent behaviors.
• Define and refine metrics for both structure and semantics, ensuring alignment with realistic use cases and operational constraints.
• Lead the development of dataset management processes and guide teams across Grafana in best practices for GenAI evaluation.

What we’re looking for:

• Experience designing and implementing evaluation frameworks for AI/ML systems.
• Familiarity with prompt engineering, structured output evaluation, and context-window management in LLM systems.
• High autonomy to collaborate and translate team goals into clear, testable criteria supported by effective tooling.
• Experience working in environments with rapid iteration and experimental development.
• A pragmatic mindset that values reproducibility, developer experience, and thoughtful trade-offs when scaling GenAI systems.
• A passion for minimizing human toil and building AI systems that actively support engineers.

In Canada, the Base compensation range for this role is CAD 153,729 - CAD 184,475 . Actual compensation may vary based on level, experience, and skillset as assessed in the interview process. Benefits include equity, bonus (if applicable) and other benefits listedhere .

*Compensation ranges are country-specific. If you are applying for this role from a different location than listed above, your recruiter will discuss your specific market’s defined pay range & benefits at the beginning of the process.

*Grafana Labs may utilize AI tools in its recruitment process to assist in matching information provided in CVs to job postings. The recruitment team will continue to review inbound CVs manually to identify alignment with current openings.

About Grafana Labs: There are more than 20M users of Grafana, the open source visualization tool, around the globe, monitoring everything from beehives to climate change in the Alps. The instantly recognizable dashboards have been spotted everywhere from a NASA launch and Minecraft HQ to Wimbledon and the Tour de France. Grafana Labs also helps more than 3,000 companies -- including Bloomberg, JPMorgan Chase, and eBay -- manage their observability strategies with the Grafana LGTM Stack, which can be run fully managed with Grafana Cloud or self-managed with theGrafana Enterprise Stack , both featuring scalable metrics (Grafana Mimir ), logs (Grafana Loki ), and traces (Grafana Tempo ).

Benefits: For more information about the perks and benefits of working at Grafana, please check out ourcareers page .

About Grafana Labs: There are more than 20M users of Grafana, the open source visualization tool, around the globe, monitoring everything from beehives to climate change in the Alps. The instantly recognizable dashboards have been spotted everywhere from a NASA launch and Minecraft HQ to Wimbledon and the Tour de France. Grafana Labs also helps more than 3,000 companies -- including Bloomberg, JPMorgan Chase, and eBay -- manage their observability strategies with the Grafana LGTM Stack, which can be run fully managed with Grafana Cloud or self-managed with theGrafana Enterprise Stack , both featuring scalable metrics (Grafana Mimir ), logs (Grafana Loki ), and traces (Grafana Tempo ). Benefits: For more information about the perks and benefits of working at Grafana, please check out ourcareers page . Equal Opportunity Employer: At Grafana Labs we’re building a company where a diverse mix of talented people want to come, stay, and do their best work. We know that our company runs on the hard work and the dedication of our passionate and creative employees. If you're excited about this role but your experience doesn’t align perfectly with every qualification in the job description, we encourage you to apply anyways.

We will recruit, train, compensate and promote regardless of race, religion, color, national origin, gender, disability, age, veteran status, and all the other fascinating characteristics that make us different and unique. We believe that equality and diversity builds a strong organization and we’re working hard to make sure that’s the foundation of our organization as we grow.

For information about how your personal data is used once you’ve applied to a job, check out our privacy policy .

Create a Job Alert

Interested in building your career at Grafana Labs? Get future opportunities sent straight to your email.

Apply for this job

indicates a required field

First Name *

Last Name *

Preferred First Name

Email *

Phone

Resume/CV

Enter manually

Accepted file types: pdf, doc, docx, txt, rtf

LinkedIn Profile *

Share a link to your LI profile.

Current company (if applicable)

Are you based and plan to work from Canada? * Select...

Are you based in Quebec? * Select...

Please share any public artifacts (code, docs, write-ups, blog posts or even social media discussions) that demonstrate your practical experience with LLMs, evaluation pipelines, or GenAI tooling. If your work is not public, a brief written summary of a key project and your role is welcome.

Are you currently eligible to work in your country of residence? * Select...

Do you now or in the future require visa sponsorship to continue working in your country of residence? * Select...

Were you referred to this role by a Grafanista? If so, let us know their name!

Anything else you'd like to share with our hiring team?

Equal Opportunity Employment Information

At Grafana Labs, we strive to ensure we grow in a way that represents the world in which we live. To help us learn more about how we can increase diversity in our candidate pool, we invite you to voluntarily provide demographic information in a confidential survey. Providing this information is optional. It will not be used in the hiring process, and has no effect on your opportunity for employment.

By voluntarily providing information and submitting your application, you explicitly consent to the collection of race, ethnicity, gender identity, and disability information and use of this information as described above.

What gender identity do you most closely identify with? * Select...

Race * Select...

Are you a person of transgender experience? * Select...