Principal AI QA Engineer — Remote (GenAI Evaluation)

BMC Software, Inc.; New York (NY); Remote; USD 153,000 - 255,000; Full time

Job summary

BMC Software, Inc. in the United States is hiring a Principal Quality Automation Engineer to shape evaluation frameworks for Generative AI and agentic AI systems. You will own the end-to-end evaluation process and build robust datasets, rubrics, and CI-tested evaluation harnesses to ensure stable, trustworthy releases.

The role emphasizes responsible-AI, governance, and cross-functional collaboration with DS, engineering, and product teams. Remote options within the United States are available.

Qualifications

• Bachelor's or master's in computer science, AI, or related field (or equivalent experience).
• Experience with Generative AI / LLM applications or AI/ML systems and quality/evaluation frameworks.

Responsibilities

• Define and implement evaluation strategies and frameworks for Generative AI, RAG, and agentic AI applications.
• Create golden datasets, rubrics, and scoring methods (including LLM-as-a-judge).
• Apply statistical sampling, repeated trials, judge calibration, dataset versioning, and anti-contamination practices.
• Stand up automated evaluation harnesses in CI so model/prompt/agent changes are re-scored before ship.
• Define quality metrics and release gates for customer-ready vs prototype.
• Design testing strategies for multi-agent workflows and AI systems using tools, APIs, MCP, and enterprise apps; Playwright where relevant.
• Evaluate end-to-end agent behaviour (planning, tool use, response generation, task completion).
• Identify failure modes using observability/tracing; recommend reliability improvements.
• Close the loop from evaluation results into model/prompt improvements with DS and AI Engineering.
• Champion Responsible-AI and human-in-the-loop principles; contribute to governance and model validation.

Skills

Python

LLM observability

CI/CD

Data-driven mindset

Education

Bachelor's or Master's in CS/AI

Tools

Langfuse

OTel

OpenSearch

Job description

BMC Software, Inc. in the United States is hiring a Principal Quality Automation Engineer to shape evaluation frameworks for Generative AI and agentic AI systems. You will own the end-to-end evaluation process and build robust datasets, rubrics, and CI-tested evaluation harnesses to ensure stable, trustworthy releases.

The role emphasizes responsible-AI, governance, and cross-functional collaboration with DS, engineering, and product teams. Remote options within the United States are available.