Senior AI QA & Evaluation Engineer

BMC Software, Inc.; United States; Remote; USD 153,000 - 255,000; Full time

Job summary

BMC Software, Inc. is seeking an experienced AI Evaluation Engineer to build and own evaluation frameworks for Generative AI and agentic AI applications.

You will create golden datasets, rubrics, and automated CI evaluation harnesses to ensure quality before shipping new features. You will collaborate across DS, engineering, and product teams, uphold responsible-AI practices, and help shape governance and model validation.

Qualifications

• Bachelor's or master's in computer science, AI, or related field.
• Experience with Generative AI / LLM applications or AI/ML systems.
• Understanding of LLM failure modes and evaluation frameworks.

Responsibilities

• Define and implement evaluation strategies and frameworks for Generative AI, RAG, and agentic AI applications.
• Create golden datasets, rubrics, and scoring methods (including LLM-as-a-judge) so quality is measured consistently.
• Stand up automated evaluation harnesses in CI so model/prompt/agent changes are re-scored before ship.
• Define quality metrics and release gates for customer-ready vs prototype.
• Design testing strategies for multi-agent workflows and AI systems.
• Evaluate end-to-end agent behaviour (planning, tool use, response generation, task completion).

Skills

Python

Generative AI / LLM apps

CI/CD

Data-driven mindset

Independent work

Education

Bachelor's or Master's in CS/AI or related field

Tools

Langfuse

OTel

OpenSearch

CI/CD tools

Job description

BMC Software, Inc. is seeking an experienced AI Evaluation Engineer to build and own evaluation frameworks for Generative AI and agentic AI applications.

You will create golden datasets, rubrics, and automated CI evaluation harnesses to ensure quality before shipping new features. You will collaborate across DS, engineering, and product teams, uphold responsible-AI practices, and help shape governance and model validation.