Senior AI Product Evaluator & Reliability Lead

Thinkbrg

United States

Remote

USD 180,000 - 250,000

Full time

8 days ago

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Berkeley Research Group (BRG) is hiring an Associate Director AI Product Engineer focused on evaluating AI in Financial Institution Operations & Technology. You will define evaluation criteria, build automated tests, and work with banking experts to ensure accuracy, reliability, and safe use of AI in banking operations.

You will lead testing across early development to live use, collaborate with engineers, and mentor colleagues.

Qualifications

Minimum 5 years of working experience with AI evaluation, including test design, scoring, and failure investigation.
Strong Python skills and experience building automated test software, connecting data sources, and recording repeatable results across product versions.
Experience in financial services or another regulated environment, translating business requirements into practical tests and controls.

Responsibilities

Define evaluation approach: what the product must get right and evidence prior to use.
Build reusable tests that run, score results, and compare versions across releases.
Reproduce failures with engineers, identify causes, verify fixes, and contribute to product improvements.
Verify accuracy by testing how well the system extracts facts and supports claims with citations.
Monitor risk: check for sensitive data exposure, guard against unsafe outputs, and document repeatable tests.