AI Engineer — LLM Production & Evaluation

LawPro.ai

Virginia

On-site

USD 150,000 - 190,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

LawPro.ai is seeking an experienced AI Engineer to own evaluation, selection, and optimization of large language models powering our data insights platform. You will ensure AI systems remain accurate, cost-efficient, and resilient as the LLM landscape evolves, managing transitions to new models and technologies.

You will perform both AI research and production engineering, benchmarking new LLMs and frameworks against our use cases, and owning end-to-end implementation with the broader team

Qualifications

  • 5+ years of AI/ML engineering in production.
  • Experience deploying LLMs on cloud infrastructure (AWS or GCP).
  • Hands-on with RAG solutions.
  • Experience with embeddings and vector databases.
  • Experience building EvalOps or Evals-as-a-Service.
  • Ability to assess model capabilities and tradeoffs.
  • Strong software engineering for production-grade pipelines.

Responsibilities

  • Continuously evaluate and optimize LLMs for accuracy, speed, and cost.
  • Develop evaluation frameworks and reduce hallucinations in legal/medical contexts.
  • Plan model transitions and integrate new models into production pipelines.
  • Implement optimizations for LLM-based orchestration pipelines.
  • Collaborate with product and GTM to communicate findings and lead implementation.
  • Own end-to-end shipping of model changes to production; tests and rollout.
  • Monitor production performance, drift, cost, and latency.
  • Document evaluation methodologies and runbooks.

Skills

AI/ML engineering
Production deployments
RAG solutions
Embeddings
Vector databases
EvalOps
LLM evaluation
Model orchestration

Tools

AWS
GCP
Docker

Job description

LawPro.ai is seeking an experienced AI Engineer to own evaluation, selection, and optimization of large language models powering our data insights platform. You will ensure AI systems remain accurate, cost-efficient, and resilient as the LLM landscape evolves, managing transitions to new models and technologies.

You will perform both AI research and production engineering, benchmarking new LLMs and frameworks against our use cases, and owning end-to-end implementation with the broader team

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior AI Engineer — LLM Evaluation & Production Systems
Senior AI Engineer — LLM Evaluation & Production Systems

LawPro.ai • Virginia (MN)

On-site
USD 140,000 - 200,000
Senior AI Engineer: LLM Evaluation, Production & Optimization
Senior AI Engineer: LLM Evaluation, Production & Optimization

LawPro.ai • North Carolina

On-site
USD 140,000 - 190,000
Senior AI Engineer — End-to-End LLM Optimization
Senior AI Engineer — End-to-End LLM Optimization

LawPro.ai • Georgia

On-site
USD 140,000 - 200,000
AI Engineer - VA
AI Engineer - VA

LawPro.ai • Virginia (MN)

On-site
USD 140,000 - 200,000
AI Engineer - NC
AI Engineer - NC

LawPro.ai • North Carolina

On-site
USD 140,000 - 190,000
AI Engineer - FL
AI Engineer - FL

LawPro.ai • Florida

On-site
USD 140,000 - 210,000
Senior AI Engineer: LLM Evaluation & Production
Senior AI Engineer: LLM Evaluation & Production

LawPro.ai • Florida

On-site
USD 140,000 - 210,000
AI Engineer - LLM Training & Evaluation (Remote)
AI Engineer - LLM Training & Evaluation (Remote)

Prolific • Tucson (AZ)

Hybrid
USD 100,000 - 130,000
Competitive pay rates
Flexible hours
Ability to work from home
AI Engineer - GA
AI Engineer - GA

LawPro.ai • Georgia

On-site
USD 140,000 - 200,000
AI Engineer - LLM Training & Evaluation (Remote)
AI Engineer - LLM Training & Evaluation (Remote)

Prolific • Memphis (TN)

Hybrid
USD 90,000 - 130,000
Competitive pay rates
Flexible hours
Ability to work from home