Test Automation Staff Engineer

Shield; San Diego (CA); On-site; USD 180,000 - 230,000; Full time

Benefits offered by this job

Medical Coverage

Parental Leave

Stock Benefits

Onsite Gym

Job summary

Shield is seeking a Staff Automated Test Engineer to provide technical leadership for automated verification across our next-generation autonomy platform.

You will define and evolve test architecture, validation infrastructure, MLOps quality systems, and CI/CD pipelines to enable Shield's software ecosystem to create, test, and deploy resilient autonomy capabilities for unmanned aircraft and robotic platforms operating in complex environments.

Qualifications

• 8+ years of software engineering, test infra, MLOps, or systems verification.
• Experience building scalable automation frameworks in Python.
• Experience validating ML systems across data, training, evaluation, deployment, and monitoring lifecycle.

Responsibilities

• Define and evolve test architecture, validation infrastructure, and CI/CD pipelines for autonomous platforms.
• Lead cross-functional verification efforts across software, ML models, simulation, and hardware-in-the-loop systems.
• Develop scalable test frameworks for autonomy software, backend services, and APIs.
• Create ML validation pipelines covering data quality, model accuracy, and deployment readiness.

Skills

Python automation

Distributed systems testing

MLOps

Test architecture

CI/CD

Education

Bachelor's degree in CS/CE

Tools

Kubeflow

Conan

Pip/Setuptools/Poetry

Nix

Job description

• The Hivemind Software Engineering Integration and Test team is seeking a Staff Automated Test Engineer to provide technical leadership for automated verification across our next-generation autonomy platform. You will define and evolve test architecture, validation infrastructure, MLOps quality systems, and CI/CD pipelines that enable the Hivemind software ecosystem to create, test, and deploy resilient autonomy capabilities for unmanned aircraft and robotic platforms operating in complex, contested, and GPS-denied environments
• You will work across production flight code, machine learning models, simulation and synthetic environments, mission planning and orchestration systems, operator-facing Ground Control Station applications, telemetry and data pipelines, cloud-native developer infrastructure, and hardware-in-the-loop systems
• In this hands-on Staff role, you will lead complex, cross-functional initiatives, establish automation and verification standards, identify systemic quality risks, and develop scalable test infrastructure. The ideal candidate combines deep Python automation and distributed-systems testing experience with technical leadership and experience validating machine learning systems throughout the data, training, evaluation, deployment, and monitoring lifecycle
• Own the technical strategy, architecture, and roadmap for automated testing, verification, and MLOps quality across the Hivemind ecosystem
• Design and maintain scalable test frameworks for autonomy software, backend services, APIs, operator-facing applications, and distributed hardware environments
• Lead functional, integration, regression, system, performance, reliability, and end-to-end testing across simulation, edge-compute, software-in-the-loop, and hardware-in-the-loop environments
• Build automated ML validation pipelines covering data quality, training reproducibility, model accuracy, robustness, regression, latency, resource utilization, and system integration
• Establish CI/CD and continuous training workflows that provide versioning and traceability for datasets, models, configurations, evaluation results, and deployment artifacts
• Develop scenario-based validation for autonomy models, including edge cases, degraded sensing or communications, distribution shifts, and representative mission conditions
• Create observability, analytics, and failure-triage capabilities for software behavior, model and data drift, inference health, test results, and production performance
• Build Python automation that improves test execution, parallelization, reporting, environment setup, experiment comparison, and developer productivity
• Create test harnesses, simulators, stubs, mocks, and synthetic data capabilities that improve system testability and coverage
• Collaborate with software, autonomy, machine learning, data, simulation, and systems engineers to define verification strategies and improve designs before implementation
• Develop and govern AI-assisted engineering workflows using coding agents and LLM-based tools for test generation, log analysis, debugging, and failure triage while maintaining security, reproducibility, and traceability

Benefits

• Excellent Medical Coverage
• Mental Health Employee Assistance Program
• Paid Parental Leave
• Pet Insurance
• Flexible Work Hours
• Onsite Gym (DC)
• Gym Discount (San Diego)
• Free Parking
• Competitive Compensation
• Stock Benefits
• 401K Services and Match

Experience qualifying integrated hardware and software systems, including automated validation of compute, GPU, storage, networking, drivers, firmware, and deployed software configurations5+ years of experience building scalable automation frameworks or developer tooling in PythonExperience testing GPU-accelerated infrastructure and workloads, including GPU scheduling, allocation, utilization, and resource contention in Kubernetes environmentsStrong understanding of asynchronous and concurrent Python programming for scalable automation and parallel test executionTypically 8+ years of relevant experience in software engineering, test infrastructure, developer tooling, MLOps, systems integration, or systems verification, or an equivalent combination of experience and demonstrated impactExperience with performance benchmarking, profiling, and observability for GPU workloads, including identifying compute, memory, storage, networking, and data-loading bottlenecksExperience validating multi-tenant Kubernetes environments, including RBAC, resource quotas, workload isolation, and scheduling behaviorExperience working in Linux-based development environmentsUnderstanding of ML quality risks such as data leakage, training-serving skew, nondeterminism, distribution shift, drift, model regression, and statistical acceptance criteriaExperience with automated observability, log collection, analytics, reporting, and root-cause analysis in complex software, data, and infrastructure systemsDemonstrated success designing test, CI/CD, or MLOps infrastructure used across multiple engineering teamsExperience validating machine learning systems across the data, training, evaluation, packaging, deployment, and monitoring lifecycleExperience developing integration and regression strategies for internally developed, third-party, open-source, and partner software, including dependency management, compatibility testing, and upgradesExperience defining model-performance baselines, automated evaluation suites, release thresholds, and candidate-to-production comparison workflowsExperience with package and dependency management, reproducible environments, and build systems such as Conan, pip, setuptools, Poetry, Nix, or similarStrong system-design skills and experience testing distributed systems, backend services, APIs, and integrated hardware and software environmentsExperience with model registries, experiment tracking, dataset or feature versioning, model serving, and automated artifact promotion using MLflow, Kubeflow, Weights & Biases, SageMaker, Vertex AI, or similar platformsExperience with GPU scheduling and orchestration platforms such as Run:ai, NVIDIA GPU Operator, KAI Scheduler, Kueue, Volcano, or similar technologiesExperience with NVIDIA GPU infrastructure, including CUDA, drivers, container runtimes, Multi-Instance GPU, GPU fractionalization, and hardware/software compatibility testingExperience with GPU profiling and performance-analysis tools such as NVIDIA Nsight, PyTorch Profiler, or similar technologiesExperience validating perception, planning, decision-making, reinforcement learning, or other autonomy models in simulation and on deployed systemsExperience testing models on embedded or edge-compute platforms, including latency, memory, power, accelerator compatibility, quantization, and hardware-specific behaviorExperience qualifying production servers or appliances, including hardware validation, burn-in, provisioning, firmware, networking, storage, and software-stack validation before deploymentExperience with reliability, fault-injection, and recovery testing across distributed compute, storage, networking, and GPU infrastructureExperience validating reproducible installation, operation, upgrades, and rollback in cloud, on-premises, disconnected, or air-gapped environmentsExperience with containers, Kubernetes, cloud infrastructure, infrastructure as code, and reproducible test environmentsProficiency with Go or TypeScript for automation tooling or UI test developmentExperience integrating Python with native C or C++ applications through bindings, wrappers, subprocess interfaces, or similar interoperability toolingAerospace, robotics, autonomy, embedded systems, or safety-critical software experienceFamiliarity with software-in-the-loop, hardware-in-the-loop, requirements-based verification, configuration management, artifact traceability, or standards such as DO-178C and MIL-STD-882