Senior AI Engineer: LLM Evaluation, Production & Optimization

LawPro.ai; North Carolina; Sur place; USD 140 000 - 190 000; Plein temps

Résumé du poste

LawPro.ai is seeking an experienced AI Engineer to own evaluation, selection, and optimization of LLMs that power our data insights platform. You will drive cost, quality, and resilience as models evolve, delivering end-to-end production-ready changes.

Responsibilities include building evaluation frameworks, benchmarking new models, and leading transitions into production while collaborating with product and operations to ensure smooth deployments.

Qualifications

• 5+ years of AI/ML engineering in production.
• Experience evaluating and deploying LLMs at scale.
• Experience building and deploying RAG solutions.
• Experience embedding models and vector DBs.
• Experience EvalOps/Evals-as-a-Service architecture.

Responsabilités

• Continuous LLM Evaluation: design and operate ongoing evaluation of LLMs across accuracy, speed, and cost.
• Eval Framework Development: build evaluation frameworks to measure LLM output quality and reduce hallucinations.
• Proactive Model Transition Planning: monitor LLM landscape and manage model replacements in production.
• AI Pipeline Optimization: optimize LLM-based orchestration pipelines end-to-end.
• Cross-Functional Collaboration: communicate findings and lead implementation with product/ops.
• End-to-End Implementation Ownership: ship model changes into production with tests and rollout.
• Operational Monitoring: implement monitoring for model performance, drift, and cost.
• Documentation: maintain runbooks and evaluation methodologies.

Connaissances

AI/ML engineering

RAG solutions

Embedding models

EvalOps

LLM orchestration

Production deployments

Cost optimization

Outils

Vector databases

LLM frameworks

Cloud platforms (AWS/GCP)

Description du poste

LawPro.ai is seeking an experienced AI Engineer to own evaluation, selection, and optimization of LLMs that power our data insights platform. You will drive cost, quality, and resilience as models evolve, delivering end-to-end production-ready changes.

Responsibilities include building evaluation frameworks, benchmarking new models, and leading transitions into production while collaborating with product and operations to ensure smooth deployments.