Senior AI Engineer — LLM Systems & Evaluation (Remote)

LawPro.ai, Inc.; Northern (KY); Hybrid; USD 150,000 - 170,000; Full time

Benefits offered by this job

Equity stake

Unlimited PTO

Health, dental, vision benefits

Remote within the US

Job summary

LawPro.ai seeks an experienced AI Engineer to own evaluation, selection, and continuous optimization of large language models powering our data insights platform for the legal sector.

You will design evaluation frameworks, manage model transitions, and optimize AI pipelines for document understanding, medical record summarization, and case chronology generation, while collaborating with product teams. This is a 100% remote role within the US with equity and strong benefits.

Qualifications

• 5+ years of AI/ML engineering experience evaluating, fine-tuning, and deploying LLMs in production environments.

Responsibilities

• Design and operate systematic evaluation processes for new LLMs across accuracy, speed, cost, and relevance.
• Build and maintain evaluation frameworks and the EvalOps culture to reduce hallucinations in medical/legal document analysis.
• Own model transitions, decommissioning stale prompts and integrating new models into production end-to-end.
• Optimize LLM-based AI pipelines for document understanding, medical record summarization, case chronology generation, and drafting support.
• Communicate evaluation findings to product/GT M stakeholders and lead technical implementations.

Skills

AI/ML engineering

LLMs in production

EvalOps

Embedding models

Vector databases

RAG solutions

Agentic workflows

Model evaluation frameworks

Python/DevOps

Tools

AWS

GCP

Job description

LawPro.ai seeks an experienced AI Engineer to own evaluation, selection, and continuous optimization of large language models powering our data insights platform for the legal sector.

You will design evaluation frameworks, manage model transitions, and optimize AI pipelines for document understanding, medical record summarization, and case chronology generation, while collaborating with product teams. This is a 100% remote role within the US with equity and strong benefits.