Senior ML Engineer - Production-Ready Defense AI (Remote)

Horizon3.ai; United States; Hybrid; USD 211,000 - 249,000; Full time

Benefits offered by this job

Inclusive team culture

Career growth opportunities

Innovative environment

Hybrid & remote work

Competitive compensation + equity

Job summary

Horizon3.ai in the United States is seeking a Senior Machine Learning Engineer on the Defensive Agent team to bring models into production and maintain them at scale across thousands of customer tenants.

You will build training and inference pipelines, optimize cost and latency, and work with AI researchers, backend, and infra engineers to productionize research.

The role requires strong Python, ML pipelines, cloud experience (AWS/Azure/GCP), and hands-on experience with model-serving infra.

Qualifications

• Bachelor's Degree in Computer Science, Computer Engineering or related field, or equivalent practical experience.
• 5+ yrs professional software engineering experience, with strong production Python.
• Demonstrated experience taking ML or LLM-backed systems from prototype to production and operating them.
• Hands-on experience with ML pipelines and tooling: training or fine-tuning workflows, experiment tracking, artifact and model registries, and reproducible data preparation.
• Experience building and operating inference or model-serving infrastructure in production, including latency and cost optimization.
• Experience building applications on cloud computing platforms such as AWS, Azure, GCP, using container technologies such as Docker and Kubernetes.
• Solid proficiency in SQL and experience with production data pipelines.

Responsibilities

• Build and own the training and post-training pipelines — data preparation, fine-tuning and preference optimization runs, experiment tracking, artifact management, and reproducibility.
• Build the inference and serving layer: model gateway with provider routing and fallback, regional pinning for data residency, batching, and caching.
• Own the release path for model-layer artifacts: version prompts, model selections, and tool definitions as deployable config; run shadow and canary deployments by tenant; make rollback fast and boring.
• Build monitoring for the model layer — behavioral drift, regression detection, output quality signals, latency, and per-tenant token and cost accounting with budget enforcement.
• Build the data and context pipelines that feed inference, including retrieval and embedding infrastructure over attack path, configuration, and remediation data, with tenant isolation enforced end to end.
• Optimize cost and latency across the inference path, and make the tradeoffs visible so product decisions are made with real numbers.
• Develop core product features in ETL and GraphQL where model outputs, run history, and evaluation results need to reach the product and internal tooling.
• Partner with researchers to move prototypes into production, and feed production constraints and failure data back into research direction.

Skills

Python

ML pipelines

Cloud platforms

Docker & Kubernetes

SQL

PostgreSQL

Neo4j

Observability tooling

Model serving

Education

Bachelor's degree in CS/CE or related

Tools

Docker

Kubernetes

GraphQL

Datadog

Prometheus

Grafana

Neo4j

PostgreSQL

Job description

Horizon3.ai in the United States is seeking a Senior Machine Learning Engineer on the Defensive Agent team to bring models into production and maintain them at scale across thousands of customer tenants.

You will build training and inference pipelines, optimize cost and latency, and work with AI researchers, backend, and infra engineers to productionize research.

The role requires strong Python, ML pipelines, cloud experience (AWS/Azure/GCP), and hands-on experience with model-serving infra.