Lead AI Engineer — LLM Evaluation & Production Optimizer

LawPro.ai; Kentucky; On-site; USD 150,000 - 190,000; Full time

Job summary

LawPro.ai is seeking an experienced AI Engineer to own evaluation, selection, and optimization of LLMs powering its data insights platform. You will design evaluation frameworks, benchmark models, and implement transitions to new technologies with end-to-end ownership.

You'll join a fast-moving team building production-grade AI systems that improve outcomes for clients, requiring deep AI and engineering intuition, systematic thinking, and strong collaboration with product and operations.

Qualifications

• 5+ years of AI/ML engineering in production environments with LLMs.
• Experience evaluating, fine-tuning, and deploying large language models at scale.
• Hands-on with RAG solutions and embedding models.
• Experience building EvalOps/Evals-as-a-Service architectures.
• Proven ability to design evaluation frameworks for model accuracy, relevancy, and hallucination detection.
• Strong software engineering background with production deployments of LLM orchestration pipelines.
• Comfort operating in fast-paced environments with ownership and bias for systematic processes.
• Excellent communication of complex results to technical and non-technical stakeholders.
• Bonus: background in medical or legal document processing; statistics or retrieval-augmented generation knowledge

Responsibilities

• Design and operate ongoing evaluation of new LLMs across accuracy, relevancy, speed and cost.
• Develop and maintain rigorous Eval frameworks and internal EvalOps culture.
• Plan and execute model transitions, including integrating new models into production.
• Implement optimizations for LLM-based orchestration pipelines for document understanding and summarization.
• Collaborate with product and GTM stakeholders to communicate results and drive implementation.
• Own end-to-end shipping of model changes into production, including deployments and validation tests.
• Monitor production model performance, track drift and cost, and report insights regularly.
• Document evaluation methodologies, comparisons, and runbooks for owned systems.

Skills

LLM production experience

RAG solutions

Embedding models

Vector databases

EvalOps / Evals

LLM orchestration

Cost and performance optimization

Production code deployment

Stakeholder communication

Medical/legal document processing (BON

Tools

AWS

GCP

Vector databases

Job description

LawPro.ai is seeking an experienced AI Engineer to own evaluation, selection, and optimization of LLMs powering its data insights platform. You will design evaluation frameworks, benchmark models, and implement transitions to new technologies with end-to-end ownership.

You'll join a fast-moving team building production-grade AI systems that improve outcomes for clients, requiring deep AI and engineering intuition, systematic thinking, and strong collaboration with product and operations.