Staff Software Engineer ML Platform (6244)

Shield; San Diego (CA); On-site; USD 140,000 - 190,000; Full time

Benefits offered by this job

Bonus

Equity

Benefits

Job summary

Shield AI, based in San Diego, seeks a Senior Software Engineer to build backend services that power ML experiment tracking, model registries and artifact management. You will connect training runs to data, code and configurations, implement APIs and robust state management, and deliver observable, upgradeable services across cloud and customer-managed environments.

You will collaborate with ML engineers to streamline their workflows, reduce manual record keeping, and ensure reliable operation

Qualifications

• Production backend services for ML platforms or data-heavy workflows.
• Strong Python or equivalent backend language.
• APIs, metadata models and persistent state design.
• Handle storage, partial failures, retries in distributed systems.
• Understand training runs, checkpoints and model versioning.
• Work with ML users to deliver maintainable software.

Responsibilities

• Build experiment-tracking and model-registry services.
• Connect dataset, code and config versions to runs.
• Develop APIs and adapters for training jobs and storage.
• Ensure recoverable workflows with retries and diagnostics.
• Expose artifacts and evaluations for reuse.
• Collaborate with ML engineers to simplify workflows.
• Deliver observability-friendly services across cloud and on-prem environments.

Skills

Python

APIs design

Distributed systems

Backend engineering

ML workflows

Tools

MLflow

Kubernetes

AWS

Job description

Job Description:

Shield AI is a venture-backed defense-tech company with the mission of protecting service members and civilians with intelligent systems. Its products include Hivemind autonomy software, V-BAT and X-BATaircraft, and Aechelon simulation and synthetic reality technologies. With offices and facilities across the U.S., Europe, the Middle East, and Asia-Pacific, Shield AI’s technology actively supports operations worldwide. For more information, visit www.shield.ai. Follow Shield AI on LinkedIn, X, Instagram, and YouTube.

Model development produces experiments, checkpoints, evaluation results and many versions of the same model. Keeping those records connected lets engineers reproduce results, compare approaches and move a trained model into use.

We are looking for a Senior Software Engineer to build the backend services that make those workflows reliable. You will connect training runs to their data, code and configuration, manage model artifacts and help machine-learning engineers spend less time reconstructing what happened in an experiment.

What you’ll do:

• Build experiment-tracking and model-registry services, integrating tools such as MLflow where appropriate.
• Connect exact dataset, code and configuration versions to training runs, checkpoints and model versions.
• Develop APIs and adapters that connect training jobs, data services and artifact storage.
• Make interrupted or failed workflows recoverable through reliable state handling, retries and useful diagnostics.
• Make checkpoints, model artifacts and evaluation results easy to find and retrieve for further training, evaluation or deployment.
• Work directly with machine-learning engineers to simplify their workflows and reduce manual record keeping.
• Deliver tested services with observability and compatible upgrades across supported cloud and customer-managed environments.

Key Outcomes:

• Engineers can trace a model version back to the data, code, configuration and run that produced it.
• Checkpoints and evaluation evidence remain available when jobs fail or workflows change.
• Developers can compare experiments and retrieve model artifacts through consistent APIs.
• Teams can reproduce results without manually reconstructing experiment history.

Required qualifications:

• Experience building production backend services for machine-learning platforms or data-intensive workflows.
• Strong software engineering skills in Python or a comparable backend language.
• Experience designing APIs, metadata models and persistent state for services used by other engineers.
• Experience handling storage, partial failures, retries and consistency in distributed systems.
• Understanding of training runs, checkpoints and model versioning.
• Ability to work directly with machine-learning users and deliver maintainable software that solves their workflow problems.

Preferred qualifications:

• Experience with MLflow, Weights & Biases, model registries or comparable experiment-management tools.
• Experience integrating Kubernetes jobs, object storage or machine-learning workflow systems.
• Experience with reproducibility, data provenance or evaluation records.
• Experience supporting open-source software or on-premises and air-gapped deployments.

Why Join Us

• The AI Factory is the foundation for how Shield AI develops and deploys autonomy.

#LI-DM2

#LF

Full-time regular employee offer package:

Pay within range listed + Bonus + Benefits + Equity

Temporary employee offer package:

Pay within range listed above + temporary benefits package (applicable after 60 days of employment)

Salary compensation is influenced by a wide array of factors including but not limited to skill set, level of experience, licenses and certifications, and specific work location. All offers are contingent on a cleared background and possible reference check. Military fellows and part‑time employees are not eligible for benefits. Please speak to your talent acquisition representative for more information.

Shield AI is proud to be an equal opportunity workplace and is an affirmative action employer. We are committed toequal employment opportunity regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, marital status, disability, gender identity or Veteran status. If you have a disability or special need that requires accommodation, please let us know.