Principal Quality Automation Engineer

BMC Software

Pune District

Sur place

INR 3 995 000 - 5 445 000

Plein temps

Il y a 3 jours
Soyez parmi les premiers à postuler

Obtenez une réponse de cet employeur

Votre CV et votre lettre de motivation, adaptés exactement à ce que l’employeur recherche.

Passez les filtres ATS

Résumé du poste

BMC Software, Inc. in India is seeking a Principal Quality Automation Engineer to design and own evaluation systems for agentic AI. You will build evaluation frameworks, golden datasets, rubrics, and CI-based eval harnesses to ensure quality before shipping.

You will define quality metrics, test multi-agent flows, and champion Responsible-AI practices while collaborating with DS, engineering, and product teams to deliver reliable, auditable AI features in a fast-paced environment.

Qualifications

  • Experience with Generative AI / LLM applications or AI/ML systems and quality/evaluation frameworks.

Responsabilités

  • Define evaluation strategies and frameworks for Generative AI, RAG, and agentic AI applications.

Connaissances

Generative AI / LLM applications
LLM failure modes understandings
Python
CI/CD integration
LLM observability/tracing
Data-driven mindset
Owns eval suite
Independent work on quality problems

Formation

Bachelor's or Master's in CS/AI or related field

Outils

Langfuse
OTel
OpenSearch

Description du poste

Principal Quality Automation Engineer - India

Country

India

State

IN_Maharashtra

City

Pune

Date Published

02-Nov-2026

Job ID

47742

Travel

up to 10%

Secondary locations

Description and Requirements

BMC empowers nearly 80% of the Forbes Global 100 to accelerate business value, faster than humanly possible. Our industry-leading portfolio unlocks human and machine potential to drive business growth, innovation, and sustainable success. BMC does this in a simple and optimized way by connecting people, systems, and data that power the world’s largest organizations so they can seize a competitive advantage.

The IZOT product line includes BMC’s Intelligent Z Optimization & Transformation products, which help the world’s largest companies to monitor and manage their mainframe systems. The modernization of mainframe is the beating heart of our product line, and we achieve this goal by developing products that improve the developer experience, the mainframe integration, the speed of application development, the quality of the code and the applications’ security, while reducing operational costs and risks. We acquired several companies along the way, and we continue to grow, innovate, and perfect our solutions on an ongoing basis.

About the Role

You build the systems that tell us whether our agentic AI is good enough to ship: evaluation frameworks, golden datasets, and rubrics; automated eval harnesses in CI; and drive down failure modes such as hallucination, drift, and unsafe or non-repeatable output. You are the reason customers can trust what our agents produce. At this level you independently own well-scoped work from definition through delivery. Scope at this level: Independently owns well-scoped eval work for a feature/team from definition through delivery. Organizational impact expected: Improves outcomes for one team or project.

Key Responsibilities
  • Define and implement evaluation strategies and frameworks for Generative AI, RAG, and agentic AI applications.
  • Create golden datasets, rubrics, and scoring methods (including LLM-as-a-judge) so quality is measured consistently.
  • Apply statistical sampling, repeated trials, judge calibration, dataset versioning, and anti-contamination practices.
  • Stand up automated evaluation harnesses in CI so model/prompt/agent changes are re-scored before ship.
  • Define quality metrics and release gates for customer-ready vs prototype.
  • Design testing strategies for multi-agent workflows and AI systems using tools, APIs, MCP, and enterprise apps; Playwright where relevant.
  • Evaluate end-to-end agent behaviour (planning, tool use, response generation, task completion).
  • Identify failure modes using observability/tracing; recommend reliability improvements.
  • Close the loop from evaluation results into model/prompt improvements with DS and AI Engineering.
  • Champion Responsible-AI and human-in-the-loop principles; contribute to governance and model validation.
Must-Have Skills & Experience
  • Bachelor's or master's in computer science, AI, or related field (or equivalent experience).
  • Experience with Generative AI / LLM applications or AI/ML systems and quality/evaluation frameworks (depth scales with level).
  • Understanding of LLM/RAG/agentic failure modes; vector DBs and embedding retrieval as level requires.
  • Python; building eval sets, rubrics, automated scoring; CI/CD integration.
  • LLM observability/tracing (e.g. Langfuse, OTel) and analytics (OpenSearch or similar) as level requires.
  • Rigorous, data-driven mindset; collaboration across DS, engineering, and product.
  • Evidence of owning an eval suite that influenced a release decision.
  • Builds consistent rubrics/datasets — not one-off manual spot checks only.
  • Works independently on scoped quality problems.
  • Evidence we will look for in hiring:
    • Past experience: Owned a defined evaluation suite or quality workflow for an AI feature and partnered with stakeholders on “good enough.”
    • Delivery evidence: Decision-ready quality report; evals integrated into a release or CI path for one product/team.
    • Shared expectation: Independently owns well-scoped work from definition through delivery.
Nice-to-Have Skills
  • Ragas, DeepEval, promptfoo, or similar eval frameworks.
  • AI safety/governance; enterprise/regulated environments.
  • OpenShift/AWS; performance/load/reliability testing of AI apps.
  • MLOps; open-weight models/local serving; MCP; Agile/Atlassian
Our commitment to you!

BMC’s culture is built around its people. We have 6000+ brilliant minds working together across the globe. You won’t be known just by your employee number, but for your true authentic self. BMC lets you be YOU!

If after reading the above, You’re unsure if you meet the qualifications of this role but are deeply excited about BMC and this team, we still encourage you to apply! We want to attract talents from diverse backgrounds and experience to ensure we face the world together with the best ideas!

BMC is committed to equal opportunity employment regardless of race, age, sex, creed, color, religion, citizenship status, sexual orientation, gender, gender expression, gender identity, national origin, disability, marital status, pregnancy, disabled veteran or status as a protected veteran. If you need a reasonable accommodation for any part of the application and hiring process, visit the accommodation request page.

BMC Software maintains a strict policy of not requesting any form of payment in exchange for employment opportunities, upholding a fair and ethical hiring process.

At BMC we believe in pay transparency and have set the midpoint of the salary band for this role at 4,700,000 INR. Actual salaries depend on a wide range of factors that are considered in making compensation decisions, including but not limited to skill sets; experience and training, licensure, and certifications; and other business and organizational needs.

The salary listed is just one component of BMC's employee compensation package. Other rewards may include a variable plan and country specific benefits.

We are committed to ensuring that our employees are paid fairly and equitably, and that we are transparent about our compensation practices.

( Returnship@BMC )

Obtenez votre examen gratuit et confidentiel de votre CV.

ou faites glisser et déposez votre fichier ici.

Similar jobs

Postes similaires à comparer

Principal Quality Automation Engineer
Principal Quality Automation Engineer

BMC Software • Maharashtra

Sur place
INR 4 200 000 - 5 800 000
Lead AI Engineer
Lead AI Engineer

BMC Software • Pune District

Sur place
INR 5 400 000 - 8 000 000
Principal AI Engineer
Principal AI Engineer

BMC Software • Bengaluru

Sur place
INR 6 500 000 - 9 500 000
Lead AI Engineer - 3 open roles
Lead AI Engineer - 3 open roles

BMC Software, Inc. • Maharashtra

Sur place
INR 5 813 000 - 7 105 000
Principal AI Engineer
Principal AI Engineer

BMC Software, Inc. • Karnataka

Sur place
INR 6 000 000 - 10 000 000
AI Engineer - Copilot studio
AI Engineer - Copilot studio

BMC Software, Inc. • Pune District

Hybride
INR 2 557 000 - 3 125 000
AI Engineer
AI Engineer

BMC Software, Inc. • Pune District

Sur place
INR 2 400 000 - 3 000 000
Lead DevOps Engineer
Lead DevOps Engineer

BMC Software • Pune District

Sur place
INR 4 000 000 - 5 400 000
Senior AI Data Architect / Solution Engineer - India based
Senior AI Data Architect / Solution Engineer - India based

BMC Software, Inc. • Bengaluru

Hybride
INR 4 200 000 - 5 200 000
Lead DevOps Engineer
Lead DevOps Engineer

BMC Software, Inc. • Pune, Pimpri-Chinchwad

Hybride
INR 4 000 000 - 5 200 000