Lead AI QA Engineer – Evaluation & Reliability

BMC Software, Inc; New York (NY); On-site; USD 153,000 - 255,000; Full time

Job summary

BMC Software, Inc. is seeking a skilled AI evaluation engineer to build evaluation frameworks, golden datasets, and rubrics for generative AI products. You will stand up automated CI evaluation harnesses and drive down hallucinations and drift while ensuring release quality.

You will own defined eval work from definition through delivery. You will collaborate with DS, engineering, and product teams, design robust testing strategies for multi-agent systems, and champion Responsible AI across the

Qualifications

• Must have a degree in computer science, AI, or related field (or equivalent experience).
• Experience with Generative AI / LLM apps or AI/ML systems and quality/evaluation frameworks.
• Understanding of LLM/RAG/agentic failure modes; vector DBs and embedding retrieval as level requires.
• Python; building eval sets, rubrics, automated scoring; CI/CD integration.
• LLM observability/tracing (e.g. Langfuse, OTel) and analytics (OpenSearch or similar).
• Data-driven mindset; collaboration across DS, engineering, and product.
• Evidence of owning an eval suite that influenced a release decision.
• Builds consistent rubrics/datasets - not one-off checks.

Responsibilities

• Define and implement evaluation strategies and frameworks for Generative AI, RAG, and agentic AI applications.
• Create golden datasets, rubrics, and scoring methods to measure quality consistently.
• Apply statistical sampling, repeated trials, judge calibration, dataset versioning, and anti-contamination practices.
• Stand up automated evaluation harnesses in CI so model/prompt/agent changes are re-scored before ship.
• Define quality metrics and release gates for customer-ready vs prototype.
• Design testing strategies for multi-agent workflows and AI systems using tools, APIs, MCP, and enterprise apps; Playwright where relevant.
• Evaluate end-to-end agent behaviour (planning, tool use, response generation, task completion).
• Identify failure modes using observability/tracing; recommend reliability improvements.
• Close the loop from evaluation results into model/prompt improvements with DS and AI Engineering.
• Champion Responsible-AI and human-in-the-loop principles; contribute to governance and model validation.

Skills

Generative AI / LLM systems

Python & CI/CD

LLM observability/tracing

Data-driven mindset

Collaboration across DS Eng Prod

Own eval suite

Education

Bachelor's or master's in CS/AI or equivalent

Tools

Langfuse

OTel

OpenSearch

CI/CD pipelines

Job description

BMC Software, Inc. is seeking a skilled AI evaluation engineer to build evaluation frameworks, golden datasets, and rubrics for generative AI products. You will stand up automated CI evaluation harnesses and drive down hallucinations and drift while ensuring release quality.

You will own defined eval work from definition through delivery. You will collaborate with DS, engineering, and product teams, design robust testing strategies for multi-agent systems, and champion Responsible AI across the