Principal ML Engineer — Production Infra & Compliance AI

blazetalent; New York (NY); Hybrid; USD 200,000 - 250,000; Full time

Benefits offered by this job

Health, dental, and vision

Equity options

Job summary

Confidential is seeking a Principal ML Engineer in New York, NY. This hybrid role focuses on leading ML infrastructure and shipping production pipelines for safe, scalable model deployment.

You will own training pipelines, evaluation harnesses, and model serving, shaping the platform used to enforce compliance before send. Strong PyTorch, distributed training, and CI/CD experience are essential.

Qualifications

• 8+ years of software engineering experience including 4+ years building infrastructure for ML or LLM systems in production.
• Hands-on depth with the modern LLM stack: PyTorch, distributed training, fine-tuning at scale (LoRA, SFT), and inference engines such as vLLM or TensorRT-LLM.
• Experience building eval harnesses, regression gates, or dataset pipelines, with a solid understanding of precision, recall, and calibration.
• Production mindset — ownership of model serving with real latency, reliability, and cost constraints.
• Strong fundamentals: Python, containers, CI/CD, cloud infrastructure, observability.
• High ownership on a small team: scope your own work, ship weekly, pragmatic build-vs-buy decisions.

Responsibilities

• Build and own training pipelines: data preparation, reproducible fine-tuning, experiment tracking, and release automation.
• Build the evaluation infrastructure: automated eval runs, regression gates, dashboards, and dataset versioning.
• Own model serving in production: low-latency inference, batching, optimization, autoscaling, and cost.
• Ship model updates safely with versioning, canarying, rollback, and drift monitoring.
• Build repeatable workflows for adapting models to new domains and customer needs.
• Turn expert labels and reviewer feedback into clean training and evaluation data.
• Set the bar for ML infrastructure as the team grows.

Skills

Software engineering

ML infrastructure

LLM systems

Python

Containers

CI/CD

Cloud infrastructure

Observability

Tools

PyTorch

Distributed training

LoRA

SFT

vLLM/TensorRT-LLM

Job description

Confidential is seeking a Principal ML Engineer in New York, NY. This hybrid role focuses on leading ML infrastructure and shipping production pipelines for safe, scalable model deployment.

You will own training pipelines, evaluation harnesses, and model serving, shaping the platform used to enforce compliance before send. Strong PyTorch, distributed training, and CI/CD experience are essential.