CVP, Model Validation & AI Governance

Burtch Works; New York (NY); Hybrid; USD 220,000 - 340,000; Full time

Job summary

Burtch Works in New York, NY is seeking a Corporate Vice President – Model Validation and AI Governance to lead validation across predictive, generative, and agentic AI solutions, partnering with Model Risk Management and Legal, Compliance, Cybersecurity, and other teams.

The role will challenge methodologies, establish robust evaluation approaches, set standards, and drive observability, dashboards, and governance practices while mentoring junior staff.

Qualifications

• Advanced degree in a quantitative field (stats, CS, data science, math, econ, eng).
• 7+ years of experience in model validation, governance, or risk management for AI/ML in regulated environments.
• Hands-on experience validating generative and agentic AI systems, including multi-step workflows and guardrails.
• Experience designing AI evaluation frameworks and testing (datasets, rubrics, adversarial testing).
• Proficiency in Python and SQL with AI tooling exposure.
• Strong communication and leadership to present findings to senior leadership.

Responsibilities

• Lead independent validation across predictive, ML, generative, and agentic AI.
• Evaluate architecture, planning, tool use, memory, and guardrails for agentic systems.
• Develop evaluation frameworks, datasets, scoring, and testing.
• Establish monitoring and third-party validation approaches.
• Evaluate vendor and foundation models with evidence and controls.
• Translate regulatory requirements into validation methodologies and SOPs.
• Present findings to senior stakeholders and governance committees.
• Create reusable templates and best practices across teams.

Skills

Statistical methods

ML validation

Python

SQL

Communication

Leadership

Education

Advanced degree in a quantitative field

Tools

Python

SQL

Job description

Job Title: Corporate Vice President – Model Validation and AI Governance

Location: New York, NY (hybrid 3 days in office)

Job Summary

The Corporate Vice President – Model Validation and AI Governance will play a key leadership role in strengthening enterprise approaches to model validation and responsible AI governance across predictive, generative, and agentic AI solutions. Working closely with Model Risk Management and partners across Artificial Intelligence & Data, Technology, Risk, Legal, Compliance, Cybersecurity, and Third-Party Risk Management, this role will translate model risk requirements into rigorous, practical validation approaches and controls.

The successful candidate will lead complex validation engagements end-to-end for internally developed and third-party solutions, independently challenging model methodologies, evaluation approaches, controls, and monitoring strategies, with a particular focus on establishing robust approaches for evaluating agentic and multi-step AI systems.

Key Responsibilities

• Lead independent validation and effective challenge across predictive, machine learning, generative AI, and agentic AI solutions, assessing model design, data and feature pipelines, methodologies, evaluation metrics, assumptions, limitations, and business impact.
• For agentic AI systems, evaluate architecture, planning and orchestration, tool use and permissions, retrieval and prompt design, memory and state, autonomy boundaries, guardrails, human-in-the-loop controls, and escalation and fallback mechanisms.
• Design and advance rigorous AI evaluation practices by developing and challenging evaluation frameworks, benchmark and golden datasets, rubric-based and LLM-as-a-judge scoring, human review protocols, regression suites, offline and online testing, and adversarial or red-team evaluations.
• Establish monitoring, observability, and third-party validation approaches that address model drift, bias and fairness, stability, hallucinations, business outcomes, and agentic failure modes.
• Evaluate vendor and foundation-model solutions through independent testing and due diligence, partnering with teams to establish tracing, logging, telemetry, dashboards, alerts, compensating controls, and audit-ready evidence.
• Translate risk and regulatory expectations into practical controls by interpreting technical standards, regulatory guidance, and internal procedures and converting requirements into validation methodologies, checklists, playbooks, standard operating procedures, monitoring expectations, and evidence requirements.
• Present validation findings, limitations, and conditions of use to senior stakeholders and governance committees.
• Set standards and strengthen validation capabilities across teams by developing reusable templates, guidelines, test harnesses, and best practices for predictive, generative, and agentic AI.
• Mentor and review the work of junior colleagues and partner across data science, engineering, product, and control functions.
• Remain current on evolving modeling techniques, AI research, evaluation methodologies, observability technologies, and governance practices.

Requirements

• Advanced degree in Statistics, Computer Science, Data Science, Mathematics, Economics, Engineering, or a related quantitative discipline.
• 7+ years of experience in model validation, model governance, or model risk management for predictive and AI/ML solutions within regulated environments.
• Deep hands-on knowledge of traditional statistical and machine learning approaches, combined with demonstrated experience validating generative and agentic AI systems, including multi-step workflows, tool use, orchestration and planning, guardrails, autonomy controls, and human oversight.
• Demonstrated experience designing, building, or independently reviewing AI evaluation frameworks, including dataset curation, benchmark design, rubric-based and LLM-as-a-judge evaluation, human review, regression testing, retrieval-quality assessment, and adversarial or red-team testing.
• Practical experience with AI/ML monitoring and observability, including tracing, logging, telemetry, dashboards, and alerting.
• Experience validating or overseeing third-party, vendor-hosted, foundation-model, or other limited-transparency AI solutions.
• Proficiency in Python and SQL, with experience using agent orchestration, evaluation, observability, or related AI tooling.
• Strong communication, leadership, and analytical skills, including the ability to interpret technical and regulatory requirements, translate them into practical controls, lead complex validations end-to-end, mentor colleagues, and present and defend technical findings with senior leaders and cross-functional governance bodies.

Preferred Qualifications

• Experience evaluating Generative AI solutions, including prompting strategies, retrieval-augmented generation (RAG), retrieval quality, model adaptation or fine-tuning trade-offs, and AI-specific privacy, security, interpretability, and stress-testing considerations.
• Knowledge of agentic AI risks and controls, including prompt injection, excessive agency, tool and permission scoping, action reversibility, and guardrail effectiveness.
• Familiarity with model and AI risk frameworks and emerging regulatory expectations, including SR 11-7, the NIST AI Risk Management Framework, the NAIC Model Bulletin on AI, and the EU AI Act.
• Experience collaborating with Legal, Compliance, Cybersecurity, and Third-Party Risk Management functions.
• Applied understanding of AI use cases within financial services or insurance, such as underwriting support, fraud detection, marketing and sales enablement, agent productivity, and customer service.