Senior Staff Backend Engineer - ML Experiment Tracking

Shield; San Diego (CA); On-site; USD 200,000 - 300,000; Full time

Benefits offered by this job

Bonus

Benefits

Equity

Job summary

Shield AI is seeking a Senior Staff Engineer to build backend services that connect training runs to data, code and configuration. You will manage model artifacts, enable reliable workflows and help ML engineers spend less time reconstructing experiment histories.

You will deliver scalable services with strong observability, integrating tools like MLflow and Kubernetes across cloud and on-prem environments. Collaboration with ML users is essential to advance autonomous systems.

Qualifications

• Experience building production backend services for machine-learning platforms or data-intensive workflows.
• Strong software engineering skills in Python or a comparable backend language.
• Experience designing APIs, metadata models and persistent state for services used by other engineers.
• Experience handling storage, partial failures, retries and consistency in distributed systems.
• Understanding of training runs, checkpoints and model versioning.
• Ability to work directly with machine-learning users and deliver maintainable software that solves their workflow problems.

Responsibilities

• Build experiment-tracking and model-registry services, integrating tools such as MLflow where appropriate.
• Connect exact dataset, code and configuration versions to training runs, checkpoints and model versions.
• Develop APIs and adapters that connect training jobs, data services and artifact storage.
• Make interrupted or failed workflows recoverable through reliable state handling, retries and useful diagnostics.
• Make checkpoints, model artifacts and evaluation results easy to find and retrieve for further training, evaluation or deployment.
• Work directly with machine-learning engineers to simplify their workflows and reduce manual record keeping.
• Deliver tested services with observability and compatible upgrades across supported cloud and customer-managed environments.

Skills

Python

Backend services

API design

Distributed systems

ML workflows

Experiment tracking

ML user collaboration

Model versioning

Tools

MLflow

Kubernetes

Cloud storage

Weights & Biases

Model registries

Job description

Shield AI is seeking a Senior Staff Engineer to build backend services that connect training runs to data, code and configuration. You will manage model artifacts, enable reliable workflows and help ML engineers spend less time reconstructing experiment histories.

You will deliver scalable services with strong observability, integrating tools like MLflow and Kubernetes across cloud and on-prem environments. Collaboration with ML users is essential to advance autonomous systems.