Senior AI Engineer — LLM Evaluation & Production Systems

LawPro.ai; Virginia (MN); On-site; USD 140,000 - 200,000; Full time

Job summary

LawPro.ai is seeking an experienced AI Engineer to own evaluation, selection, and optimization of large language models powering its data insights platform. You will benchmark models, design EvalOps, and manage end-to-end transitions into production with rigorous validation.

You will collaborate with product and operations, implement monitoring, and document methodologies. The role demands deep AI and engineering intuition, strong ownership, and the ability to translate evaluation findings into

Qualifications

• 5+ years of AI/ML engineering in production evaluating and deploying LLMs.
• Experience deploying models to AWS or GCP at scale.
• Hands-on with RAG, embedding models, and vector databases.

Responsibilities

• Design and operate systematic LLM evaluation across tasks.
• Build and maintain EvalOps frameworks for LLMs.
• Plan and execute model transitions and production migrations.
• Optimize AI pipelines for cost, quality, and latency.
• Own end-to-end deployment and testing of model changes.
• Collaborate with product and GTM stakeholders.
• Implement monitoring and observability for production models.
• Document methodologies, results, and runbooks.
• Ensure compliance with high-stakes domains (legal/medical).

Skills

AI/ML engineering

Production deployment

RAG solutions

Embedding models

Vector databases

EvalOps / Evals-as-a-Service

LLM orchestration

Model evaluation frameworks

Cost optimization

Cross-functional collaboration

Tools

AWS

GCP

Python

LLM tooling

Job description

LawPro.ai is seeking an experienced AI Engineer to own evaluation, selection, and optimization of large language models powering its data insights platform. You will benchmark models, design EvalOps, and manage end-to-end transitions into production with rigorous validation.

You will collaborate with product and operations, implement monitoring, and document methodologies. The role demands deep AI and engineering intuition, strong ownership, and the ability to translate evaluation findings into