Threat Investigation Engineer

10a Labs; United States; Remote; USD 160,000 - 200,000; Full time

Job summary

10a Labs is hiring a Threat Investigation Engineer for remote, 1-year contract. You will design, develop, and maintain backend services and AI-assisted applications to support threat investigation workflows.

You will work directly with investigators to translate needs into practical engineering solutions, build AI-powered features, and ensure reliable production systems. The role requires strong Python backend development skills, experience with production operations, and the ability to

Qualifications

• Strong software engineering experience, particularly in Python and backend application development.
• Experience with platform engineering, site reliability engineering (SRE), or production operations.
• Experience building, deploying, debugging, and maintaining production software systems.
• Familiarity with REST APIs, backend services, asynchronous processing, and system integrations.
• Practical experience working with AI-powered applications, LLM APIs, or model-assisted development tools.
• Strong troubleshooting and debugging skills, with the ability to investigate unfamiliar systems and resolve complex production issues.
• Experience supporting production environments, including monitoring, incident response, and reliability improvements.
• Ability to independently scope technical problems, prioritize work, and deliver maintainable solutions.
• Strong communication skills and experience collaborating with technical and nontechnical stakeholders.
• Sound judgment when handling sensitive information, operational incidents, and time-sensitive technical challenges.
• Availability to participate in scheduled on-call coverage, including weekends and holidays.

Responsibilities

• Design, develop, test, and maintain backend services and AI-assisted applications that support threat investigation workflows.
• Partner directly with investigators to understand operational challenges, identify opportunities for improvement, and translate requirements into practical engineering solutions.
• Build and integrate AI-powered features, APIs, and services that improve the efficiency, accuracy, and reliability of investigation processes.
• Troubleshoot and resolve production issues, including failed workflows, processing delays, data inconsistencies, and system errors.
• Participate in rotating on-call coverage, including weekends and holidays, to diagnose issues, implement fixes, and elevate when necessary.
• Improve system reliability and observability through monitoring, logging, automated testing, documentation, and operational tooling.
• Investigate complex technical issues across backend services, data pipelines, and AI-assisted workflows, identifying root causes and implementing sustainable solutions.
• Collaborate with engineering and platform teams to deploy improvements, maintain system stability, and support evolving operational needs.
• Own engineering tasks end-to-end, from understanding user requirements and designing solutions to implementation, testing, deployment, and maintenance.

Skills

Python backend

Platform engineering/SRE

Production operations

REST APIs

Asynchronous processing

Debugging on-call support

Stakeholder collaboration

Tools

REST APIs

Async processing

Distributed systems

Job description

About 10a Labs: 10a Labs is the safety and threat-intelligence layer trusted by frontier AI labs, AI unicorns, Fortune 10 companies, and leading global technology platforms. Our adversarial red teaming, model evaluations, and intelligence collection enable engineering, safety, and security teams to stay ahead of evolving threats and deploy AI systems safely.

Remote | 1-year Contract

About the Role

We are seeking a Threat Investigation Engineer to build, improve, and support AI-powered systems used in high-stakes safety and threat investigations.

This is a hands-on software engineering role combining backend development, production troubleshooting, and platform reliability. You'll work closely with investigators and engineering teams to develop practical solutions, improve existing workflows, and ensure critical investigation systems operate reliably.

The ideal candidate is a strong Python engineer with experience building and maintaining production applications, troubleshooting complex systems, and integrating AI-powered capabilities. You should be comfortable navigating ambiguous technical challenges, working directly with end users, and taking ownership of solutions from initial problem definition through deployment and ongoing support.

This position includes participation in a rotating on-call schedule, including weekend and holiday coverage, to support critical investigation workflows.

What You'll Do

• Design, develop, test, and maintain backend services and AI-assisted applications that support threat investigation workflows.
• Partner directly with investigators to understand operational challenges, identify opportunities for improvement, and translate requirements into practical engineering solutions.
• Build and integrate AI-powered features, APIs, and services that improve the efficiency, accuracy, and reliability of investigation processes.
• Troubleshoot and resolve production issues, including failed workflows, processing delays, data inconsistencies, and system errors.
• Participate in rotating on-call coverage, including weekends and holidays, to diagnose issues, implement fixes, and elevate when necessary.
• Improve system reliability and observability through monitoring, logging, automated testing, documentation, and operational tooling.
• Investigate complex technical issues across backend services, data pipelines, and AI-assisted workflows, identifying root causes and implementing sustainable solutions.
• Collaborate with engineering and platform teams to deploy improvements, maintain system stability, and support evolving operational needs.
• Own engineering tasks end-to-end, from understanding user requirements and designing solutions to implementation, testing, deployment, and maintenance.

What We're Looking For

• Strong software engineering experience, particularly in Python and backend application development.
• Experience with platform engineering, site reliability engineering (SRE), or production operations.
• Experience building, deploying, debugging, and maintaining production software systems.
• Familiarity with REST APIs, backend services, asynchronous processing, and system integrations.
• Practical experience working with AI-powered applications, LLM APIs, or model-assisted development tools.
• Strong troubleshooting and debugging skills, with the ability to investigate unfamiliar systems and resolve complex production issues.
• Experience supporting production environments, including monitoring, incident response, and reliability improvements.
• Ability to independently scope technical problems, prioritize work, and deliver maintainable solutions.
• Strong communication skills and experience collaborating with technical and nontechnical stakeholders.
• Sound judgment when handling sensitive information, operational incidents, and time-sensitive technical challenges.
• Availability to participate in scheduled on-call coverage, including weekends and holidays.

Nice to Have

• Experience with asynchronous job processing, background workers, task queues, or distributed systems.
• Familiarity with cloud infrastructure and services such as AWS, GCP, or Azure.
• Experience building internal tools, workflow automation, or investigator-facing applications.
• Familiarity with observability tools, structured logging, alerting, and incident management practices.
• Experience working in cybersecurity, trust and safety, threat intelligence, or other high-stakes operational environments.

Additional Information

• Employment Type: 1 year contract, 40 hours full-time
• Compensation: $160,000 - $200,000
• Location: Remote
• Schedule: Includes rotating on-call coverage, with weekend and holiday availability required.

This role supports sensitive safety and threat investigations and may involve exposure to potentially distressing case information.