Senior AI Engineer: LLM Evaluation, Production & Optimization

LawPro.ai; North Carolina; On-site; USD 140,000 - 190,000; Full time

Job summary

LawPro.ai is seeking an experienced AI Engineer to own evaluation, selection, and optimization of LLMs that power our data insights platform. You will drive cost, quality, and resilience as models evolve, delivering end-to-end production-ready changes.

Responsibilities include building evaluation frameworks, benchmarking new models, and leading transitions into production while collaborating with product and operations to ensure smooth deployments.

Qualifications

• 5+ years of AI/ML engineering in production.
• Experience evaluating and deploying LLMs at scale.
• Experience building and deploying RAG solutions.
• Experience embedding models and vector DBs.
• Experience EvalOps/Evals-as-a-Service architecture.

Responsibilities

• Continuous LLM Evaluation: design and operate ongoing evaluation of LLMs across accuracy, speed, and cost.
• Eval Framework Development: build evaluation frameworks to measure LLM output quality and reduce hallucinations.
• Proactive Model Transition Planning: monitor LLM landscape and manage model replacements in production.
• AI Pipeline Optimization: optimize LLM-based orchestration pipelines end-to-end.
• Cross-Functional Collaboration: communicate findings and lead implementation with product/ops.
• End-to-End Implementation Ownership: ship model changes into production with tests and rollout.
• Operational Monitoring: implement monitoring for model performance, drift, and cost.
• Documentation: maintain runbooks and evaluation methodologies.

Skills

AI/ML engineering

RAG solutions

Embedding models

EvalOps

LLM orchestration

Production deployments

Cost optimization

Tools

Vector databases

LLM frameworks

Cloud platforms (AWS/GCP)

Job description

LawPro.ai is seeking an experienced AI Engineer to own evaluation, selection, and optimization of LLMs that power our data insights platform. You will drive cost, quality, and resilience as models evolve, delivering end-to-end production-ready changes.

Responsibilities include building evaluation frameworks, benchmarking new models, and leading transitions into production while collaborating with product and operations to ensure smooth deployments.