End-to-End AI Engineer for Production LLMs

LawPro.ai; United States; On-site; USD 140,000 - 230,000; Full time

Job summary

LawPro.ai is seeking an experienced AI Engineer to own evaluation, selection, and optimization of LLMs powering its data insights platform. You will manage transitions to new models, ensuring cost, quality, and resilience in a fast-moving landscape.

You will do both AI research and production engineering, building evaluation frameworks, benchmarking models, and implementing production-ready changes with end-to-end ownership.

Qualifications

• 5+ years of AI/ML engineering experience evaluating, fine-tuning, and deploying large language models in production environments.
• Hands-on development and implementation of multiple RAG solutions.
• Hands-on experience leveraging embedding models and vector databases.
• Hands-on experience building agentic workflows and EvalOps architectures.
• Deep familiarity with the LLM ecosystem and model routing, cost, quality, speed tradeoffs.
• Proven experience designing evaluation frameworks for LLM output quality (legal/medical or similar).
• Strong software engineering foundation with production deployments of LLM orchestration frameworks.

Responsibilities

• Continuous LLM evaluation and benchmarking across accuracy, speed, and cost.
• Build and maintain evaluation frameworks and EvalOps culture.
• Plan and execute model transitions, integrating new models into production.
• Optimize AI pipelines for document understanding and analysis tasks.
• Collaborate with product and GTM to communicate findings and implement changes.
• Own end-to-end shipping of model changes into production with tests and rollouts.
• Implement monitoring and observability for model performance and cost.
• Document methodologies, results, and runbooks.

Skills

LLM evaluation

RAG solutions

Vector databases

EvalOps

Production engineering

Model deployment

Observability

Documentation

Tools

AWS

GCP

Job description

LawPro.ai is seeking an experienced AI Engineer to own evaluation, selection, and optimization of LLMs powering its data insights platform. You will manage transitions to new models, ensuring cost, quality, and resilience in a fast-moving landscape.

You will do both AI research and production engineering, building evaluation frameworks, benchmarking models, and implementing production-ready changes with end-to-end ownership.