AI Engineer

LawPro.ai, Inc.; Northern (KY); Hybrid; USD 150,000 - 170,000; Full time

Benefits offered by this job

Equity stake

Unlimited PTO

Health, dental, vision benefits

Remote within the US

Job summary

LawPro.ai seeks an experienced AI Engineer to own evaluation, selection, and continuous optimization of large language models powering our data insights platform for the legal sector.

You will design evaluation frameworks, manage model transitions, and optimize AI pipelines for document understanding, medical record summarization, and case chronology generation, while collaborating with product teams. This is a 100% remote role within the US with equity and strong benefits.

Qualifications

• 5+ years of AI/ML engineering experience evaluating, fine-tuning, and deploying LLMs in production environments.

Responsibilities

• Design and operate systematic evaluation processes for new LLMs across accuracy, speed, cost, and relevance.
• Build and maintain evaluation frameworks and the EvalOps culture to reduce hallucinations in medical/legal document analysis.
• Own model transitions, decommissioning stale prompts and integrating new models into production end-to-end.
• Optimize LLM-based AI pipelines for document understanding, medical record summarization, case chronology generation, and drafting support.
• Communicate evaluation findings to product/GT M stakeholders and lead technical implementations.

Skills

AI/ML engineering

LLMs in production

EvalOps

Embedding models

Vector databases

RAG solutions

Agentic workflows

Model evaluation frameworks

Python/DevOps

Tools

AWS

GCP

Job description

About LawPro.ai

LawPro.ai is a pioneering legal technology company transforming the personal injury law sector with its AI-powered platform. Our solution automates the tedious process of medical record review, generating detailed treatment chronologies, identifying red flags, and even accelerating demand letter creation. With new features like Case Assistant, we're helping firms increase case value while reducing manual review time.

At the heart of LawPro.ai is a Large Language Model tailored to the legal industry, designed to enhance efficiency, improve case outcomes, and enable firms to scale. The platform is fully HIPAA-compliant and made by practitioners for practitioners. Backed by The LegalTech Fund and Scopus Ventures, we're building at speed and scale. Join us at an exciting stage of growth where your work will directly impact how justice is delivered.

About the Role

We are looking for an experienced AI Engineer to own the evaluation, selection, and continuous optimization of the large language models and AI processes that power LawPro.ai's data insights and analytics platform. You will ensure our AI systems remain accurate, cost-effective, and resilient as the LLM landscape evolves, proactively managing transitions to new models and technologies. You will build the solutions and processes needed to raise our bar for cost, quality, and resilience.

This role blends AI research and production engineering: staying ahead of a fast-moving model landscape, benchmarking new LLMs, techniques, and frameworks against our use cases, and owning both the recommendation and the implementation. We value engineers who bring deep AI and engineering intuition alongside a systematic, process-driven mindset, people who can design evaluation frameworks, interpret model behavior, and carry changes into production without relying on others to finish the work.

You'll be a key contributor to a fast-moving team building production-grade AI systems that materially impact how law firms optimize outcomes for their clients.

What You'll Do

• Continuous LLM Evaluation: Design and operate a systematic process to evaluate new and emerging LLMs across accuracy, relevancy, speed, and cost, benchmarking continuously against tasks in our orchestration pipeline.
• Eval Framework Development: Build and maintain evaluation frameworks and pioneer our internal EvalOps culture, measuring output accuracy, relevance, and faithfulness, with a focus on reducing hallucinations in medical record summarization and legal document analysis.
• Model Transition Ownership: Monitor the LLM landscape for deprecation timelines and replacement models, then own execution end to end, integrating new models into production, adjusting for model behavior, and decommissioning stale or underperforming prompts and endpoints.
• AI Pipeline Optimization: Implement optimizations to LLM-based orchestration pipelines for document understanding, medical record summarization, case chronology generation, and drafting support, owning code changes, deployments, and validation with a bias toward surgical execution over wholesale refactors.
• Cross-Functional Collaboration: Communicate model evaluation findings to product and GTM stakeholders and lead the technical implementation yourself, ensuring clean handoffs from discovery through staging to production.
• Operational Monitoring: Implement monitoring and observability for model performance, benchmarking output and cost, detecting drift, and reporting to management on an ongoing basis.
• Documentation: Maintain documentation of evaluation methodologies, model comparisons, transition decisions, and runbooks for systems you own.

Who You Are

• 5+ years of AI/ML engineering experience evaluating, fine-tuning, and deploying LLMs in production environments, including cloud infrastructure (AWS or GCP) at scale and writing production-deployed LLM orchestration frameworks and multi-model pipelines.
• Hands-on development of multiple RAG solutions.
• Hands-on experience with embedding models and vector databases.
• Hands-on experience building agentic workflows and implementing EvalOps or Evals-as-a-Service architecture.
• Deep familiarity with the LLM ecosystem, able to critically assess model capabilities, limitations, and fit, including heuristic-gated model routing and cost, quality, speed, and capability tradeoffs.
• Proven experience designing evaluation frameworks for LLM output quality in high-stakes domains (legal, medical, or similar), including hallucination detection.
• Comfort in a fast-paced, high-ambiguity environment, with strong ownership and a bias for systematic process-building over one-off fixes.
• Excellent communication skills; able to translate complex model evaluation findings into clear recommendations for technical and non-technical stakeholders.
• Bonus: experience with unstructured medical or legal document processing, or a background in classical ML (statistics, embeddings, RAG).

What We Offer & Perks

• Competitive compensation, $150,000 - $170,000 base salary
• Equity stakes with opportunity to share in the company upside
• 100% remote within the US, with flexible arrangements
• Unlimited paid time off
• Comprehensive health, dental, and vision benefits
• Strong growth potential and advancement opportunities as LawPro.ai scales

LawPro.ai is an Equal Opportunity Employer. We hire based on merit and qualifications alone, without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, age, disability, veteran status, or any other characteristic protected by applicable law.