Lead, Legal AI Benchmarking & Evaluation

Newcode.ai; New York (NY); On-site; USD 120,000 - 190,000; Full time

Job summary

Newcode.ai in New York is seeking a highly analytical professional with a strong statistical background to join our Head of Evaluations. In this role, you will design, implement, and scale the testing frameworks used to evaluate our platform, ensuring our AI products meet the highest standards of legal reasoning, factual accuracy, and regulatory compliance while maintaining a near-zero hallucination rate.

You will design benchmarks for contract drafting, information extraction, legal research,

Qualifications

• Strong statistical background and analytical mindset.
• Experience designing evaluation benchmarks for AI/ML models.

Responsibilities

• Design Legal Benchmarks for: Contract Drafting, Information Extraction, Legal Research, and Contract Review
• Build, source and maintain relevant datasets
• Audit AI Output: Review and score complex AI-generated legal text, contract analyses, and statutory interpretations for accuracy and precision and lay out a strategy.
• Define Evaluation Metrics: Establish clear criteria for grading model performance, specifically focusing on logical reasoning, citation accuracy, and the model's ability to safely abstain from answering.
• Collaborate with Engineering: Partner directly with Engineering to translate legal errors into actionable technical feedback for model fine-tuning

Skills

Analytical skills

Statistical background

Attention to detail

Education

PhD or Masters in statistics, mathematics, ML or equivalent

Tools

Python

Pandas

NumPy

Jupyter notebooks

Job description

Newcode.ai in New York is seeking a highly analytical professional with a strong statistical background to join our Head of Evaluations. In this role, you will design, implement, and scale the testing frameworks used to evaluate our platform, ensuring our AI products meet the highest standards of legal reasoning, factual accuracy, and regulatory compliance while maintaining a near-zero hallucination rate.

You will design benchmarks for contract drafting, information extraction, legal research,