AI Engineer — LLM Evaluation & Production Systems

LawPro.ai; Ohio; On-site; USD 120,000 - 160,000; Full time

Job summary

LawPro.ai is seeking an experienced AI Engineer to own evaluation, selection, and continuous optimization of large language models and AI processes powering our data insights platform. You will ensure AI systems are accurate, cost-effective, and resilient as the LLM landscape evolves, proactively managing transitions to new models and technologies.

You will perform both AI research and production engineering, benchmarking new LLMs and techniques, and owning both recommendation and implementation

Qualifications

• 5+ years of AI/ML engineering experience deploying LLMs in production.
• Experience building and deploying models to AWS or GCP at scale.
• Experience with RAG solutions.
• Experience with embedding models and vector databases.
• Experience designing evaluation frameworks for LLM output quality.

Responsibilities

• Design and operate an ongoing process to evaluate new LLMs across accuracy, relevancy, speed, and cost.
• Build and maintain rigorous evaluation frameworks (Evals) and EvalOps culture.
• Plan and execute model transitions, integrating new models into production pipelines.
• Implement optimizations to LLM-based orchestration pipelines for document understanding and drafting support.
• Collaborate with product and GTM stakeholders and own technical implementations.
• Take end-to-end ownership for shipping model changes into production and monitoring.

Skills

AI/ML engineering

RAG systems

Embedding models

EvalOps / Evals

LLM orchestration

Production deployments

Cloud AWS/GCP

Data pipelines

Cross-functional collaboration

Monitoring & observability

Tools

Vector databases

LLM frameworks

Cloud platforms (AWS/GCP)

Job description

LawPro.ai is seeking an experienced AI Engineer to own evaluation, selection, and continuous optimization of large language models and AI processes powering our data insights platform. You will ensure AI systems are accurate, cost-effective, and resilient as the LLM landscape evolves, proactively managing transitions to new models and technologies.

You will perform both AI research and production engineering, benchmarking new LLMs and techniques, and owning both recommendation and implementation