AI Product Evaluation Lead - Banking Ops (Remote)

Berkeley Research Group, LLC; Emeryville (CA); Remote; USD 180,000 - 250,000; Full time

Job summary

BRG is seeking an Associate Director AI Product Engineer for Financial Institution Operations & Technology. This remote role focuses on hands-on evaluation of AI products used in banking, designing tests, and providing evidence for release decisions.

You will work with banking experts and engineers to define expected results, measure performance, and drive product improvements across development and live use. A minimum of 5 years in AI evaluation and strong Python skills are required.

Qualifications

• Minimum 5 years of working experience with a minimum of 1 year experience in AI evaluation.
• Demonstrated ownership of AI evaluations that informed real release decisions, including test design, expert-reviewed answers, scoring, and failure investigation.
• Strong Python skills and experience building automated test software, connecting data sources, and recording repeatable results across product versions.
• Ability to select representative samples, interpret error rates and uncertainty, and distinguish genuine improvement from weak tests or inconsistent scoring.
• Experience building or improving AI products used in operations, with practical controls for sensitive data, output accuracy, and human review.

Responsibilities

• Define what each product must get right, how errors could affect customers or operations, and what evidence is needed before use.
• Build reusable software that runs tests, scores results, and compares versions. Make testing part of everyday development, not a final checkpoint.
• Work with engineers to reproduce failures, identify causes, and verify fixes. Contribute product code where needed to improve accuracy and control.
• Set pass criteria and scoring rules with business and risk teams. Distinguish minor defects from errors that must block release.
• Maintain representative test cases, including incomplete records and rare, high-impact errors. Reserve unseen examples for final testing to avoid overstating performance.
• Have banking experts establish expected answers and resolve scoring disagreements. Measure factual accuracy, missing information, incorrect additions, and support from cited sources.

Skills

AI evaluation

Python

Automated testing

Data validation

Release risk

Job description

BRG is seeking an Associate Director AI Product Engineer for Financial Institution Operations & Technology. This remote role focuses on hands-on evaluation of AI products used in banking, designing tests, and providing evidence for release decisions.

You will work with banking experts and engineers to define expected results, measure performance, and drive product improvements across development and live use. A minimum of 5 years in AI evaluation and strong Python skills are required.