Engineer, AI – AI Evaluation & Model Risk Lead

BrickRed Systems; Bellevue (WA); On-site; USD 190,000 - 230,000; Full time

Job summary

BrickRed Systems seeks an Engineer, AI - AI Evaluation & Model Risk Lead to steer how AI models are evaluated, cleared, monitored, and documented for enterprise use. You will own model-risk clearance, ongoing evaluation, model cards, evaluation harnesses, and test suites to ensure models are measured and current.

The ideal candidate has hands-on ML experience, Python, cloud computing, data analysis and modeling, prompt engineering, and a strong grasp of model evaluation.

Qualifications

• 2+ years of professional AI/ML or data-focused experience.
• Bachelor's degree or higher in CS/Engineering/AI/Data Science or related field.
• 2–4+ years with ML models, tuning, and prompt engineering.
• 2–4+ years collaborating with cross-functional teams to deploy AI.
• Strong knowledge of cloud computing and data modeling.
• Experience documenting model risk, cards, and governance.

Responsibilities

• Own initial model-risk clearance and approved-use ratings for AI models.
• Lead ongoing evaluation and re-clearance after model changes.
• Evaluate models against live traffic and detect drift.
• Author and maintain model cards documenting tests and risks.
• Build and run evaluation harnesses and test suites.
• Collaborate with Cyber and Responsible AI teams on security and governance.
• Integrate AI systems with cross-functional teams and stakeholders.
• Develop, test, debug, document, and communicate development activities.
• Validate results with user representatives and support solution integration.
• Select and reuse or create components to improve efficiency and quality.
• Review unit tests, scenarios, and test plans; ensure quality improvements.
• Contribute to architecture artifacts (HLD/LLD/SAD) and data models.

Skills

Machine Learning

Python

Data Analysis

Data Modeling

Prompt Engineering

Model Evaluation

Cross-functional Collaboration

Analytical Thinking

Education

Bachelor's degree in Computer Science, Engineering, Artificial Intelligence, Data Science, or related field

Advanced degree (MS/PhD) preferred

Tools

Python

R

Cloud Platforms

Job description

We are seeking an experienced Engineer, AI - AI Evaluation & Model Risk Lead to lead how AI models are evaluated, cleared, monitored, and documented for enterprise use. This role owns model-risk clearance, ongoing model evaluation, model cards, evaluation harnesses, and test suites to ensure AI models are measured, documented, and current.

The ideal candidate will have hands-on experience with machine learning models, AI tools, Python, cloud computing, data analysis, data modeling, model tuning, prompt engineering, and model evaluation. You will collaborate with Safety Standards, Cyber, Responsible AI, and cross-functional teams to support the safe and effective adoption of AI solutions.

Key Responsibilities

• Own initial model-risk clearance and approved-use ratings that gate AI models into use.
• Own ongoing model evaluation and re-clearance following model version changes.
• Evaluate models against live traffic and identify model drift.
• Author and maintain model cards documenting what was tested, approved use cases, and identified risks.
• Build and run AI evaluation harnesses and test suites aligned with established Safety Standards.
• Collaborate with Cyber teams responsible for security red-teaming and Responsible AI teams.
• Collaborate with cross-functional teams to ensure effective integration and functionality of AI systems.
• Develop, test, debug, document, and communicate application and component development activities.
• Validate technical results with user representatives and support overall solution integration.
• Select appropriate technical solutions by reusing, improving, reconfiguring, or creating components.
• Optimize application development for efficiency, cost, and quality.
• Review and create unit test cases, scenarios, test plans, and execution results.
• Contribute to HLD, LLD, SAD, architecture, application features, business components, and data models.
• Perform defect root-cause analysis and mitigation and identify proactive quality improvements.
• Support release processes, configuration management, documentation, project delivery, and technical presentations.
• Interface with customers and development teams to clarify requirements and provide technical guidance.

Required Qualifications

• 2+ years of professional experience in relevant AI, machine learning, software development, or data-focused roles.
• Bachelor's degree with 3 years of related work experience, OR advanced degree with 1 year of related work experience, OR equivalent combination of education and experience.
• Degree in Computer Science, Engineering, Artificial Intelligence, Data Science, or related field preferred.
• 2-4+ years of experience developing and deploying machine learning models, including fine-tuning and prompt engineering.
• 2-4+ years of experience with AI tools and software development using Python or R.
• 2-4+ years of experience collaborating with cross-functional teams to integrate AI solutions into business workflows.
• Strong knowledge of Cloud Computing.
• Strong experience in Data Analysis, Data Management, and Data Modeling.
• Knowledge of Model Tuning and Prompt Engineering.
• Strong understanding of Model Evaluation and AI model-risk concepts.
• Experience creating and maintaining Model Cards & Documentation.
• Understanding of Model Risk & Clearance processes.
• Strong analytical and problem-solving abilities with the ability to work under pressure and manage multiple tasks.

Preferred Qualifications

• Experience with AI model-risk evaluation and clearance.
• Experience building and maintaining AI evaluation harnesses and test suites.
• Experience evaluating models against production/live traffic and identifying model drift.
• Experience with model cards, model documentation, and AI governance practices.
• Experience with machine learning fine-tuning and prompt engineering.
• Preferred certifications include:
• Certified Analytics Professional (CAP)
• Certified Data Scientist (CDS)

ABOUT BRICKRED SYSTEMS

BrickRed Systems is a global leader in next-generation technology consulting and workforce solutions, specializing in delivering high-quality talent across digital, engineering, marketing, analytics, finance, operations, and business transformation domains. With a strong emphasis on innovation, scalability, and client success, BrickRed Systems helps organizations solve complex business challenges by providing skilled professionals across strategy, technology, creative, and operational functions. BrickRed fosters a culture of continuous learning, collaboration, and excellence, enabling professionals to contribute to high-impact global initiatives while advancing their careers.