Senior ML Engineer, MLOps & AI Evaluation

Ellkay, Llc; Indiana; On-site; USD 150,000 - 210,000; Full time

Job summary

Ellkay, LLC is seeking a Senior ML Engineer - MLOps and AI Evaluation to build the shared ML operations and evaluation foundation for our classical ML, LLM, and agent services. You will implement reproducible training and deployment pipelines, versioned models, and prompts.

This role focuses on monitoring model quality, latency, cost, and safety, with cross-functional collaboration across security, data platform, and product teams, to ensure scalable, compliant AI workflows.

Qualifications

• Build reproducible training and deployment pipelines with versioned models and prompts.
• Create synthetic fixtures and benchmarks for healthcare data.
• Deploy serving pipelines with monitoring for latency, drift and cost.
• Implement evaluation gates and automated judges against SME labels.
• Collaborate with Security and Data Platform on PHI-safe telemetry and data handling.
• Own resource optimization and reproducible model-training jobs.
• Work with product engineers, clinical SMEs, and QA teams.
• Follow platform release workflows for model proposals.

Responsibilities

• Build ML infrastructure for MLOps and AI evaluation.
• Maintain lineage and reproducible release evidence.
• Validate automated judges against SME labels.
• Monitor model quality, latency, and unit cost.
• Collaborate with cross-functional teams on data governance and security.

Skills

Python

ML Engineering

MLOps

CI/CD

Model Deployment

Telemetry/Monitoring

Education

Bachelor's degree in CS or related field

Tools

Docker

Kubernetes

Git

Terraform

MLFlow

Job description

Ellkay, LLC is seeking a Senior ML Engineer - MLOps and AI Evaluation to build the shared ML operations and evaluation foundation for our classical ML, LLM, and agent services. You will implement reproducible training and deployment pipelines, versioned models, and prompts.

This role focuses on monitoring model quality, latency, cost, and safety, with cross-functional collaboration across security, data platform, and product teams, to ensure scalable, compliant AI workflows.