AI Evaluation Infrastructure Consultant

Landing Point; Village of Pelham (NY); On-site; USD 152,000 - 179,000; Full time

Job summary

Landing Point seeks an AI Evaluation Infrastructure Consultant to ensure the quality and compliance of AI systems across production environments. The role builds evaluation disciplines and tooling, develops golden test sets, and integrates drift detection to support scalable governance.

In collaboration with risk, compliance, and legal teams, you will establish production-readiness gates, run regression and adversarial testing, and provide evaluation evidence for governance and third-party

Qualifications

• 7+ years of experience in software quality, data science, machine learning, or related fields.
• Experience designing evaluation methods for LLM or ML systems.
• Strong understanding of testing methodology and statistical rigor.
• Experience with bias and fairness evaluation; familiarity with fair-lending concepts is a plus.
• Ability to define and enforce production-readiness gates and acceptance criteria.
• Experience partnering with risk, compliance, and legal teams.

Responsibilities

• Develop evaluation harnesses and tooling for AI systems.
• Create golden test sets and scenario libraries for expected behaviors and edge cases.
• Conduct regression testing to identify quality changes.
• Perform hallucination and grounding/faithfulness testing.
• Conduct bias and fairness testing to support fair-lending obligations.
• Implement adversarial and red-team testing.
• Ensure policy-adherence testing against compliance and regulatory requirements.
• Monitor drift detection and ongoing production.
• Manage human and subject-matter-expert evaluation workflows.
• Establish production-readiness gates for AI systems.
• Provide evaluation evidence and reporting for AI governance.
• Define AI vendor acceptance criteria for third-party solutions.

Skills

LLM evaluation methods

Statistical testing

Bias and fairness evaluation

Production-readiness gates

Risk & compliance collaboration

Job description

Company Overview:

A leading financial services company is seeking an AI EvaluationInfrastructure Consultant toensure the quality and compliance of AI systems. This role is crucial for building the evaluation discipline and tooling necessary for AI systems to be accurate, safe, and ready to scale.

Job Responsibilities:

• Develop evaluation harnesses and tooling for AI systems.
• Create golden test sets and scenario libraries for expected behaviors and edge cases.
• Conduct regression testing to identify quality changes.
• Perform hallucination and grounding/faithfulness testing.
• Conduct bias and fairness testing to support fair-lending obligations.
• Implement adversarial and red-team testing.
• Ensure policy-adherence testing against compliance and regulatory requirements.
• Monitor drift detection and ongoing production.
• Manage human and subject-matter-expert evaluation workflows.
• Establish production-readiness gates for AI systems.
• Provide evaluation evidence and reporting for AI governance.
• Define AI vendor acceptance criteria for third-party solutions.

Qualifications:

• 7+ years of experience in software quality, data science, machine learning, or related fields.
• Experience designing evaluation methods for LLM or ML systems.
• Strong understanding of testing methodology and statistical rigor.
• Experience with bias and fairness evaluation; familiarity with fair-lending concepts is a plus.
• Ability to define and enforce production-readiness gates and acceptance criteria.
• Experience partnering with risk, compliance, and legal teams.

Compensation:

Pay Rate: $120/hr, DOE