AI Evaluation & Model Risk Lead

BrickRed Systems; Bellevue (WA); On-site; USD 190,000 - 230,000; Full time

Job summary

BrickRed Systems seeks an Engineer, AI - AI Evaluation & Model Risk Lead to steer how AI models are evaluated, cleared, monitored, and documented for enterprise use. You will own model-risk clearance, ongoing evaluation, model cards, evaluation harnesses, and test suites to ensure models are measured and current.

The ideal candidate has hands-on ML experience, Python, cloud computing, data analysis and modeling, prompt engineering, and a strong grasp of model evaluation.

Qualifications

• 2+ years of professional AI/ML or data-focused experience.
• Bachelor's degree or higher in CS/Engineering/AI/Data Science or related field.
• 2–4+ years with ML models, tuning, and prompt engineering.
• 2–4+ years collaborating with cross-functional teams to deploy AI.
• Strong knowledge of cloud computing and data modeling.
• Experience documenting model risk, cards, and governance.

Responsibilities

• Own initial model-risk clearance and approved-use ratings for AI models.
• Lead ongoing evaluation and re-clearance after model changes.
• Evaluate models against live traffic and detect drift.
• Author and maintain model cards documenting tests and risks.
• Build and run evaluation harnesses and test suites.
• Collaborate with Cyber and Responsible AI teams on security and governance.
• Integrate AI systems with cross-functional teams and stakeholders.
• Develop, test, debug, document, and communicate development activities.
• Validate results with user representatives and support solution integration.
• Select and reuse or create components to improve efficiency and quality.
• Review unit tests, scenarios, and test plans; ensure quality improvements.
• Contribute to architecture artifacts (HLD/LLD/SAD) and data models.

Skills

Machine Learning

Python

Data Analysis

Data Modeling

Prompt Engineering

Model Evaluation

Cross-functional Collaboration

Analytical Thinking

Education

Bachelor's degree in Computer Science, Engineering, Artificial Intelligence, Data Science, or related field

Advanced degree (MS/PhD) preferred

Tools

Python

R

Cloud Platforms

Job description

BrickRed Systems seeks an Engineer, AI - AI Evaluation & Model Risk Lead to steer how AI models are evaluated, cleared, monitored, and documented for enterprise use. You will own model-risk clearance, ongoing evaluation, model cards, evaluation harnesses, and test suites to ensure models are measured and current.

The ideal candidate has hands-on ML experience, Python, cloud computing, data analysis and modeling, prompt engineering, and a strong grasp of model evaluation.