Senior Perception & Tracking Engineer

Allen Control Systems; Austin (TX); On-site; USD 140,000 - 190,000; Full time

Benefits offered by this job

Competitive salary

Health insurance

Paid Time Off

Job summary

Allen Control Systems is seeking a Senior/Staff Perception and Tracking Engineer to advance real-time multi-target tracking for autonomous robotics. You will design and implement 3D computer vision pipelines, data association, and track maintenance for resource-constrained hardware.

Work with robotics, software, and platform engineers to deploy deep learning-based perception, optimize for low latency, and validate systems on embedded GPUs.

Qualifications

• Bachelor’s or Master’s degree in CS, aerospace, applied math or related field.
• 2–3 years of experience with perception algorithms, robotics, or ML, focusing on 3D CV tracking.
• Strong experience developing vision-based tracking for real-time applications.
• Experience implementing object detection (YOLO, SSD, R-CNN) and tracking algorithms (Kalman/particle filters, ByteTrack, DeepSORT).
• Solid applied math foundation in linear algebra, probability, statistics, and estimation.

Responsibilities

• Design, implement, test, and maintain core 3D computer vision based multi-object tracking algorithms and infrastructure.
• Develop and optimize CV algorithms for real-time detection and tracking of small objects in robotics systems.
• Implement and improve object detection and multi-object tracking pipelines using modern deep learning techniques.
• Train, evaluate, and deploy machine learning models for perception tasks in dynamic environments.
• Design image preprocessing and feature extraction pipelines to improve model robustness and performance.
• Optimize tracking pipelines for low-latency, real-time performance on embedded hardware and GPUs.
• Collaborate with robotics and software teams to integrate perception systems into autonomous platforms.
• Participate in full software development lifecycle with testing and evaluation frameworks.
• Define tracking-specific metrics, evaluation harnesses, and test scenarios to improve accuracy and robustness.

Skills

3D Computer Vision tracking

Python

C++

Linear algebra / math

Education

Bachelor’s or Master’s in Computer Science / Aerospace Engineering / Applied Math or related field

Tools

YOLO

SSD

R-CNN

Kalman filters

ByteTrack

DeepSORT

CUDA

Job description

Company Overview

Allen Control Systems (ACS) is a cutting-edge defense startup founded by two former Navy electrical engineers with a proven track record in robotics and software. We are developing an autonomous gun turret using advanced computer vision and control systems to precisely detect, track, and neutralize enemy drones.

With an engineering-first culture, ACS values technical excellence and innovation. Backed by our founders’ successful exits from two previous ventures acquired for a combined $180M in 2022, we are committed to ensuring that the groundbreaking technologies we develop have a real-world impact.

Position Overview

We are seeking a Senior/Staff Perception and Tracking Engineer to join our team. This person will specialize in deep algorithmic understanding and production-grade software development for real-time multi-target inertial tracking for robotics and autonomous systems. In this role, you will design and implement both classical and modern deep learning algorithms for real-time perception algorithms that enable robots and autonomous systems to detect, track, and understand objects in dynamic environments. You will work closely with robotics, software, and platform engineers to develop high-performance computer vision and tracking systems that operate reliably in real-world conditions and on resource-constrained hardware. This role is ideal for engineers who enjoy solving challenging perception problems, building real-time systems, and working at the intersection of machine learning, robotics, and computer vision.

We’re hiring across multiple experience levels, including:

• Associate Perception & Tracking Engineer
• Perception & Tracking Engineer
• Senior Perception & Tracking Engineer

What You’ll Do:

• Design, implement, test, debug, and maintain core 3D Computer Vision based multi-object tracking algorithms and infrastructure, including measurement conditioning, data association, track correlation, propagation, track initialization/maintenance/deletion, and handling of track uncertainty.
• Develop and optimize computer vision algorithms for real-time detection and tracking of small objects in robotic and autonomous systems.
• Implement and improve object detection and multi-object tracking pipelines using modern deep learning techniques. Analyze and resolve complex tracking issues such as track fragmentation, ID switches, poor association in clutter, sensitivity to measurement quality/latency, and coordinate system challenges.
• Train, evaluate, and deploy machine learning models for perception tasks in dynamic environments.
• Design image preprocessing and feature extraction pipelines to improve model robustness and performance.
• Optimize tracking pipelines for low-latency, real-time performance on embedded hardware and/or GPUs. Profile bottlenecks and apply algorithmic or implementation improvements.
• Collaborate with robotics and software teams to integrate perception systems into autonomous platforms and real-world deployments.
• Participate in the full software development lifecycle: requirements, design, implementation, unit/integration/system testing, deployment, and maintenance. Follow strong software engineering practices and support automated testing and evaluation frameworks.
• Define tracking-specific metrics, evaluation harnesses, and test scenarios. Quantify and improve accuracy, robustness, stability, and computational efficiency.

Required Technical Skills:

• 2-3 years of experience working with perception algorithms, robotics, or machine learning and comfortable in a role that specializes in 3D Computer Vision based tracking
• Bachelor’s or Master’s degree in Computer Science, Aerospace Engineering, Applied Math or related field
• Strong experience developing vision-based tracking algorithms for real-time applications
• Experience implementing object detection (YOLO, SSD, R-CNN, or similar) and classic or deep learning-based tracking algorithms (Kalman and particle filters, ByteTrack, or similar)
• Solid applied math foundation in linear algebra, probability, statistics, and optimal estimation (Kalman filtering and variants)
• Experience with image preprocessing, feature extraction, and other classical computer vision pipelines
• Strong programming skills in Python and C++

Preferred Technical Skills:

• Production experience integrating real-time tracking algorithms into robotic perception systems on edge or embedded hardware.
• Experience with multi-sensor fusion and integrating heterogeneous sensor data (cameras + LiDAR, RADAR, or other modalities) into tracking pipelines.
• Ability to go beyond library usage (e.g., ByteTrack, DeepSORT) and implement or significantly customize core tracking logic.
• Familiarity with practical tracking challenges and mitigations: measurement latency, varying update rates, coordinate transformations/registration, track-to-track fusion concepts, IMM-style maneuver modeling, and handling of redundant or conflicting measurements.
• Experience working on robotics systems or embedded vision platforms
• Experience optimizing vision workloads using CUDA or GPU acceleration

What We Offer

• Competitive salary
• Health, Dental, Vision Insurance
• Paid Time Off

Allen Control Systems is an Equal Opportunity Employer, providing equal employment opportunities to all employees and applicants for employment. Allen Control Systems prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws.