Principal ML Engineer - Production-Scale AI Infra

blazetalent; New York (NY); Hybrid; USD 200,000 - 250,000; Full time

Benefits offered by this job

Health insurance

Dental insurance

Vision insurance

Equity

Hybrid work

Job summary

blazetalent in New York, NY is seeking a Principal ML Engineer to own end-to-end ML infrastructure for production models. You will build training pipelines, evaluation tooling, model serving, and release processes on a small, senior team in a hybrid work setting.

The role emphasizes ownership, scalability, cost-aware deployments, and collaboration with research counterparts to push industrial-grade ML in a regulated or high-stakes domain.

Qualifications

• 8+ years of software engineering experience, with 4+ years building ML/LLM infra in production.
• Hands-on depth with modern LLM stack, distributed training, and scalable inference.
• Experience building eval harnesses, regression gates, and dataset pipelines.
• Proven track record owning model serving under latency, reliability, and cost constraints.
• Strong Python, containers, CI/CD, cloud infra, and observability.

Responsibilities

• Build and own training pipelines: data prep, reproducible fine-tuning, experiment tracking, and release automation.
• Build evaluation infrastructure: automated eval runs, regression gates, dashboards, dataset versioning.
• Own model serving in production: low-latency inference, batching, optimization, autoscaling, cost management.
• Ship model updates safely with versioning, canarying, rollback, and drift monitoring.
• Build repeatable workflows for adapting models to new domains and customer needs.
• Convert expert labels and reviewer feedback into clean training and evaluation datasets.
• Set the technical bar for ML infrastructure as the team grows.

Skills

Python

PyTorch

LLM stack

CI/CD

Cloud infra

Education

BS/MS in CS or related

Tools

vLLM

TensorRT-LLM

LoRA/SFT

Job description

blazetalent in New York, NY is seeking a Principal ML Engineer to own end-to-end ML infrastructure for production models. You will build training pipelines, evaluation tooling, model serving, and release processes on a small, senior team in a hybrid work setting.

The role emphasizes ownership, scalability, cost-aware deployments, and collaboration with research counterparts to push industrial-grade ML in a regulated or high-stakes domain.