Senior Quality & AI Evaluation Engineer (AIRE)

BMW TechWorks India Private Limited

Pune District, Bengaluru

On-site

INR 3,000,000 - 4,200,000

Full time

9 days ago

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

BMW TechWorks India Private Limited is seeking a Senior Quality & AI Evaluation Engineer to drive automated testing for the AIRE tool and its AI features. You will design test strategies, implement CI/CD test suites, and evaluate LLM-based components with rigorous data-driven methods.

You will collaborate with teams in Pune or Bengaluru to ensure robust quality gates while aligning with automotive standards and project timelines.

Qualifications

  • Degree in Computer Science or related field or equivalent experience.
  • Minimum of five years of test or quality engineering experience.
  • Experience automating tests for data-intensive systems.

Responsibilities

  • Define and maintain the test and evaluation strategy with levels and gates.
  • Automate BDD scenarios from acceptance criteria and contract tests.
  • Run load tests and report results against quality goals.
  • Build/run evaluations for AI agents and functions with the Lead Engineer.
  • Produce test evidence for tool qualification alongside the Safety Specialist.
  • Approve changes to test data and evaluation datasets.
  • Set up automated quality gates from the start and keep them mandatory.

Skills

Test automation
CI/CD
Python
BDD
Contract testing
LLM evaluation

Education

BCA
B.Tech/B.E
BSc

Tools

pytest
Gherkin
Hypothesis
k6/Locust
OpenTelemetry

Job description

Senior Quality & AI Evaluation Engineer ...

BTWI/SQ-AEE/1922027

  • Digital Company
  • Pune OR Bengaluru

End Date 31 Dec 2026

Required Experience 5 - 8 years

Skills
Skill

BCA B.Tech/B.E BSc

Job Description
Role

Senior Quality & AI Evaluation Engineer (Senior)

AIRE (AI-native Requirements Engineering) is a new BMW Group tool for managing vehicle requirements, the descriptions of what a car and its software must do. It keeps the full history of every requirement, shows how requirements are linked to each other and to tests, and shows what is affected when something changes. It helps teams meet automotive standards such as Automotive SPICE and ISO 26262. AIRE is built for AI agents first and a user interface later, and the team uses AI agents to help build it. In this role you make sure AIRE and its AI features work correctly, through automated tests and checks.

Location

Pune or Bangalore

Number of openings

1

What awaits you/ Job Profile
  • Define and maintain the test and evaluation strategy, including test levels, quality gates and evidence.
  • Automate BDD scenarios from the acceptance criteria, contract tests and property-based tests.
  • Run load tests against the quality goals and report the results.
  • Build and run evaluations for agents and AI functions together with the Lead Agentic Engineering & AI Platform Engineer.
  • Produce test evidence for tool qualification together with the Functional Safety & Process Compliance Specialist.
  • Act as the human approver for changes to test oracles, evaluation datasets and quality-gate thresholds.
  • Set up automated quality gates (tests, coverage, static analysis, performance and security tests) from the very beginning, and keep them mandatory for every change.
What should you bring along
  • Degree in Computer Science or a related field, or equivalent relevant experience.
  • Minimum of five years of test or quality engineering experience.
  • Demonstrated experience automating tests for data-intensive systems.
Must have technical skill
  • Strong test engineering experience in Python (for example pytest), including test automation in CI/CD.
  • Experience with BDD (for example Gherkin, behave or pytest-bdd) and contract testing.
  • Experience with property-based testing (for example Hypothesis) for complex data logic.
  • Experience with load and performance testing (for example k6 or Locust) against defined service-level objectives.
  • Experience evaluating LLM-based systems and agents with golden datasets, regression evaluations and acceptance thresholds.
  • Proficient AI-native engineering practice, including using agents to generate tests while keeping test oracles and evaluation criteria under human control.
Good to have technical skills
  • Knowledge of ISO 26262 Part 8 tool qualification and of test evidence for Automotive SPICE.
  • Experience with mutation testing and test-quality metrics.
  • Experience with observability-based testing (for example OpenTelemetry).
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Integration & Ingestion Engi...
Senior Data Integration & Ingestion Engi...

BMW TechWorks India Pvt Ltd. • Pune District

On-site
INR 1,200,000 - 1,800,000
Lead Agentic Engineering & AI Platform E...
Lead Agentic Engineering & AI Platform E...

BMW TechWorks India Pvt Ltd. • Pune District

On-site
INR 2,500,000 - 4,500,000
AI Senior QA
AI Senior QA

Hiver • Bengaluru

On-site
INR 1,500,000 - 2,200,000
Test Engineer - Automotive Systems & AI-Assisted Test Automation
Test Engineer - Automotive Systems & AI-Assisted Test Automation

BMW TechWorks India Private Limited • Pune

On-site
INR 700,000 - 1,100,000
Lead Value Flow Engineer (AIRE)
Lead Value Flow Engineer (AIRE)

BMW TechWorks India Pvt Ltd. • Pune District

On-site
INR 4,000,000 - 8,000,000
Senior Security & IAM Engineer (AIRE)
Senior Security & IAM Engineer (AIRE)

BMW TechWorks India Pvt Ltd. • Pune District

On-site
INR 1,200,000 - 1,800,000
AI Evaluation Engineer - Proofline
AI Evaluation Engineer - Proofline

Fermi AI • Bengaluru

On-site
INR 1,500,000 - 2,100,000
AI Senior QA
AI Senior QA

Hiver, Inc. • Bengaluru

On-site
INR 2,200,000 - 3,400,000
Senior AI QA Engineer (Pune )
Senior AI QA Engineer (Pune )

Money Forward India • Pune District

On-site
INR 2,800,000 - 4,000,000
Lead Software & Systems Architect AIRE
Lead Software & Systems Architect AIRE

BMW TechWorks India Pvt Ltd. • Pune District

On-site
INR 4,000,000 - 7,000,000