AI Engineering Lead, Evals and Systems Integration

Bain & Company, Inc.; Warszawa; On-site; PLN 350,000 - 550,000; Full time

Benefits offered by this job

401(k) matching

Generous vacation and benefits

Job summary

Bain & Company, Inc. is seeking an AI Engineering Lead, Evals and Systems Integration to own the evaluation framework for a multi-module Growth AI Performance System.

You will coordinate across Data Ontology, Agentic Core, and Platform teams, ensuring quality gates before deployment and working directly with consulting teams post-live deployments. The role combines the rigour of enterprise-grade AI evaluation with the speed of startup-style iteration, offering real influence over architecture

Qualifications

• Experience designing and running evaluations for AI systems
• Strong Python proficiency and CI/CD familiar
• Ability to articulate evaluation criteria and metrics
• Experience with data ontology and knowledge graphs

Responsibilities

• Design and own evaluation framework across the Growth AI platform
• Evaluate integration points between data ontology and agentic core
• Coordinate evaluation work across distributed teams
• Present findings to senior stakeholders
• Ensure security/compliance in evaluation infrastructure

Skills

LLM evaluation

Agentic AI

Python

Systems integration

Education

Bachelor's degree in CS/Engineering

Tools

Knowledge graph

Ontology

Job description

AI Engineering Lead, Evals and Systems Integration

Job ID

111330

Work Areas

Technology & Engineering

Employment Type

Permanent Full-Time

Location(s)

Boston, Chicago, London, Sao Paulo, Warsaw

Description & Requirements

What Makes Us a Great Place to Work

We are proud to be consistently recognized as one of the world’s best places to work. We are currently the top ranked consulting firm on Glassdoor’s Best Places to Work list and have earned the #1 overall spot a record seven times. Extraordinary teams are at the heart of our business strategy, but these don’t happen by chance. They require intentional focus on bringing together a broad set of backgrounds, cultures, experiences, perspectives, and skills in a supportive and inclusive work environment. We hire people with exceptional talent and create an environment in which every individual can thrive professionally and personally.

Who You’ll Work With

You will be part of a team building Bain’s integrated suite of agentic decisioning tools that gives large organizations always‑on intelligence across their most important growth decisions. The role sits at an intersection few organizations can offer: the analytical depth and client access of a top‑tier consultancy, the technical ambition of a product company, and engineering work at the frontier of applied AI.

You will work inside Bain’s standing product team, building software that helps the world’s largest companies answer the questions that drive their business: where to invest, where to focus, and how to win. The impact of what you build is direct and immediate, and the platform compounds in value with every release.

The platform has two layers working together: an agentic core and a set of decision modules, each solving a distinct commercial problem. The build lifecycle is connected to the real world: as case teams work alongside clients across industries and geographies, that feedback flows straight back to engineering and into the product. Every client engagement makes the platform smarter, broader, and more resilient.

This is a startup inside an established company. You get genuine early‑stage ownership and real influence over architecture and product direction, backed by Bain’s global scale, client relationships, and leadership commitment; the pace of a startup with the runway of a global firm.

We are looking for engineering leads who see how to make the platform’s evaluations sharper and bring that instinct to every sprint without being asked, and who are comfortable with problems that do not have a known answer. The work involves building evaluation frameworks for LLM‑based and agentic systems, systems integration testing, and data problems with no off‑the‑shelf answer. Experience with agentic systems matters: tool and context protocols such as MCP, evaluation harnesses, and entity resolution across messy enterprise data. This is applied AI in production, at scale, with real users and a team depending on how you approach problems that do not have a known answer. The field is moving quickly and the team is built to move with it, so the willingness to keep learning counts for as much as what you already know.

Where You’ll Fit Within the Team

The AI Engineering Lead, Evals and Systems Integration, owns the evaluation framework and systems integration layer across the entire Growth AI Performance System platform. This is a cross‑cutting role: you are not evaluating a single agent or a single module, but the coherence, quality, and reliability of the full system: from the data ontology layer through the Agentic Core to the three product modules that clients use. Your work is the quality gate that determines what is ready to ship and what is not.

You will work directly across three teams: the Data Ontology and Knowledge team, where you evaluate whether the semantic layer is producing outputs that are accurate, consistent, and consumable by downstream AI systems; the Agentic Core and Interface team, where you evaluate whether agentic workflows are executing reliably, reasoning correctly, and failing gracefully; and the Platform and Security team, where you ensure that evaluation and observability infrastructure is hardened, compliant, and integrated with the platform’s security and audit controls.

You will work across the platform and product modules focused on marketing ROI, brand, and pricing strategy, defining what good looks like for each domain, building the evaluation datasets and rubrics that capture it, and enforcing deployment‑readiness standards consistently across all three modules before anything reaches a live client engagement.

This role has significant client proximity. You will engage with consulting teams after client deployments to understand where the system fell short, translate that feedback into structured evaluation improvements, and present quality and reliability findings to senior client stakeholders. The evaluation framework you build is not just an internal engineering tool: it is part of what makes clients trust what the platform produces.

Evaluation framework design and ownership

• Own and evolve the evaluation framework across the full Growth AI Performance System platform, defining quality standards, scoring rubrics, evaluation datasets, and deployment‑readiness thresholds for the data ontology layer, the Agentic Core, and all three product modules
• Design and maintain evaluation datasets and regression test suites that enable systematic, reproducible measurement of system quality across the full stack: from knowledge graph accuracy and entity resolution correctness through to agentic workflow reliability and end‑to‑end output quality
• Define what ‘good enough to deploy’ means at each layer of the system (semantic layer, agentic orchestration, retrieval pipelines, and product module outputs), and enforce those standards consistently before any change reaches a live client environment
• Design experiments to measure system quality, detect regressions, and surface the root cause of quality failures; apply statistical reasoning and experimental design to distinguish signal from noise in evaluation results
• Build automated evaluation pipelines that integrate into CI/CD workflows so that quality gates run before every deployment; ensure evaluation is a continuous process, not a pre‑release checkpoint

Systems integration and cross-team evaluation

• Evaluate the integration points between the data ontology layer and the Agentic Core, validating that knowledge graph outputs are correctly structured, semantically consistent, and reliably consumable by agentic retrieval and reasoning systems
• Work with the Data Ontology and Knowledge team to define evaluation criteria for entity resolution quality, taxonomy consistency, and data contract correctness; build the tests and monitoring that catch ontology‑layer failures before they propagate downstream
• Work with the Agentic Core and Interface team to evaluate multi‑agent orchestration, tool use reliability, context passing, and end‑to‑end workflow correctness; define failure mode taxonomies and ensure the system degrades gracefully when components fail
• Work with the Platform and Security team to ensure evaluation infrastructure is integrated with the platform’s security controls, audit logging, and compliance requirements; validate that evaluation pipelines themselves handle sensitive client data correctly
• Coordinate evaluation work across distributed engineering teams; translate evaluation findings into concrete, prioritized engineering improvements and communicate quality status clearly to product and engineering leadership
• Engage with consulting teams after live client deployments to understand where system outputs fell short, tracing quality failures through the evaluation framework, identifying root causes, and feeding structured findings back to the relevant engineering teams
• Present evaluation findings, quality metrics, and system reliability assessments to senior stakeholders including executive leadership; translate technical quality measures into business terms that explain what the platform can and cannot be trusted to do
• Contribute to internal stakeholder where the platform’s evaluation approach and quality standards are explained; build confidence in the rigor of the platform’s deployment process through credible, evidence‑based communication
• Stay current with emerging evaluation methodologies, responsible AI assessment frameworks, and AI‑specific risk and safety approaches; lead knowledge sharing on evaluation best practices across the engineering team

About You

Technical

• Deep experience designing and running evaluations for LLM‑based and agentic AI systems across the full delivery lifecycle, with a track record of building evaluation frameworks that actually change what ships
• Strong understanding of systems integration evaluation, able to design tests that validate not just individual components but the correctness and coherence of the interfaces between them, including knowledge graph outputs, API contracts, and agentic context passing
• Strong Python proficiency; experience building automated evaluation pipelines, regression test suites, and scoring rubrics that integrate into CI/CD workflows and run without manual intervention
• Statistical reasoning and experimental design: able to design experiments that measure system quality rigorously, interpret results without over‑reading noise, and surface root causes of quality failures to engineering teams
• Familiarity with knowledge graph and ontology evaluation, understanding what entity resolution correctness, taxonomy consistency, and semantic layer quality mean in practice and how to measure them programmatically
• Experience with human‑in‑the‑loop annotation and feedback systems; familiarity with preference learning, data labelling strategies, and feedback collection methodologies for LLM and agentic applications
• Understanding of security, audit, and compliance requirements for evaluation infrastructure: how to handle sensitive client data in evaluation pipelines, what audit trails need to capture, and how eval integrates with platform security controls
• Native AI fluency: active use of AI‑assisted coding tools (Cursor, Claude Code, or equivalent) and demonstrated ongoing development of evaluation methodology as the field evolves
• Demonstrated commitment to responsible AI practices: risk assessment, bias and fairness evaluation, AI safety principles, and familiarity with AI‑specific risks including prompt injection, data leakage, and agentic workflow exploitation

Experience

• 8 or more years in AI engineering, ML engineering, or applied AI roles with significant hands‑on building responsibilities
• Track record of building and owning evaluation frameworks for production AI or ML systems, not just contributing to evaluation, but defining the standards, building the infrastructure, and enforcing deployment gates
• Experience evaluating systems that span multiple components and integration points, not just model‑level evaluation but system‑level evaluation across data pipelines, APIs, and agentic workflows
• Experience working with or alongside knowledge graph, ontology, or semantic data teams is a strong advantage
• Direct experience engaging with clients or senior business stakeholders in a technical capacity, presenting quality findings, advising on system reliability, or contributing to client‑facing delivery
• Experience working in agile, cross‑functional product or consulting‑adjacent environments; strong troubleshooting and root cause analysis skills

Education

Associate’s degree / Bachelor’s degree in computer science, engineering, mathematics, or a related technical field; or an equivalent combination of education, training, and experience

U.S. Compensation Information

Compensation for this role includes base salary, annual discretionary performance bonus, 401(k) plan with an annual employer contribution based on years of service and Bain’s best in class benefits package (details listed below).

Some local governments in the United States require a good‑faith, reasonable salary range to be included in job postings for open roles. The estimated annualized compensation for this role is as follows:

• In Massachusetts, the good‑faith, reasonable annualized full‑time salary range for this role is between $198,250 – $237,500; placement within this range will vary based on several factors including, but not limited to experience, education, licensure/certifications, training and skill level
• In Illinois, the good‑faith, reasonable annualized full‑time salary range for this role is between $189,500 – $227,000; placement within this range will vary based on several factors including, but not limited to experience, education, licensure/certifications, training and skill level
• Annual discretionary performance bonus
• This role may also be eligible for other elements of discretionary compensation
• 4.5% 401(k) company contribution, which increases after 3 years of service and is 100% vested upon start date

Bain & Company’s comprehensive benefits and wellness program is designed to help employees achieve personal independence, protection and stability in the areas most important to you and your family.

• Bain pays 100% individual employee premiums for medical, dental and vision programs, offering one of the most comprehensive medical plans for employees without impacting your paycheck
• Generous paid time off, including parental leave, sick leave and paid holidays
• Fully vested 401(k) company contribution
• Paid Life and Long‑Term Disability insurance

For all other locations, the good‑faith, reasonable annualized full‑time salary range for this role is commensurate with competitive geographic market rates for this role and will vary based on several factors including, but not limited to experience, education, licensure/certifications, training and skill level.

It is unlawful in Massachusetts to require or administer a lie detector test as a condition of employment or continued employment. An employer who violates this law shall be subject to criminal penalties and civil liability.