Threat Investigation Engineer

10alabs; United States; Remote; USD 160,000 - 200,000; Full time

Job summary

10a Labs is seeking a Threat Investigation Engineer (Backend/Platform) for a remote, 1-year contract. You will build, enhance, and support AI-powered investigation systems, focusing on reliability and practical security workflows.

The role blends backend development with platform reliability, requiring hands-on coding, troubleshooting, and collaboration with investigators and engineering teams. On-call coverage, including weekends and holidays, is required.

Qualifications

• Strong Python engineering and backend development experience.
• Experience with platform engineering, SRE, or production operations.
• Experience building, deploying, debugging, and maintaining production software systems.
• Familiarity with REST APIs, backend services, and asynchronous processing.
• Practical experience with AI-powered applications, LLM APIs, or model-assisted tools.
• Strong troubleshooting and debugging skills for unfamiliar systems.
• Experience supporting production environments, including monitoring and incident response.
• Ability to scope technical problems, prioritize work, and deliver maintainable solutions.

Responsibilities

• Design, develop, test, and maintain backend services and AI-assisted applications that support threat investigation workflows.
• Collaborate with investigators to understand operational challenges and translate requirements into engineering solutions.
• Build and integrate AI-powered features, APIs, and services to improve efficiency and reliability of investigations.
• Troubleshoot and resolve production issues, including failed workflows and data inconsistencies.
• Participate in rotating on-call coverage, including weekends and holidays.
• Improve system reliability and observability through monitoring, logging, and automated testing.
• Investigate complex issues across backend services and data pipelines; identify root causes.
• Collaborate with platform teams to deploy improvements and maintain stability.
• Own engineering tasks end-to-end from requirements through deployment and maintenance.

Skills

Python

Backend development

Platform engineering

SRE

Production operations

REST APIs

Asynchronous processing

AI-powered applications

Troubleshooting

Incident response

Tools

AWS

GCP

Azure

Job description

Threat Investigation Engineer (Backend/Platform)

Remote | 1-year Contract

About 10a Labs: 10a Labs is the safety and threat-intelligence layer trusted by frontier AI labs, AI unicorns, Fortune 10 companies, and leading global technology platforms. Our adversarial red teaming, model evaluations, and intelligence collection enable engineering, safety, and security teams to stay ahead of evolving threats and deploy AI systems safely.

About the Role

We are seeking a Threat Investigation Engineer to build, improve, and support AI-powered systems used in high-stakes safety and threat investigations.

This is a hands-on software engineering role combining backend development, production troubleshooting, and platform reliability. You'll work closely with investigators and engineering teams to develop practical solutions, improve existing workflows, and ensure critical investigation systems operate reliably.

The ideal candidate is a strong Python engineer with experience building and maintaining production applications, troubleshooting complex systems, and integrating AI-powered capabilities. You should be comfortable navigating ambiguous technical challenges, working directly with end users, and taking ownership of solutions from initial problem definition through deployment and ongoing support.

This position includes participation in a rotating on-call schedule, including weekend and holiday coverage, to support critical investigation workflows.

What You'll Do

• Design, develop, test, and maintain backend services and AI-assisted applications that support threat investigation workflows.
• Partner directly with investigators to understand operational challenges, identify opportunities for improvement, and translate requirements into practical engineering solutions.
• Build and integrate AI-powered features, APIs, and services that improve the efficiency, accuracy, and reliability of investigation processes.
• Troubleshoot and resolve production issues, including failed workflows, processing delays, data inconsistencies, and system errors.
• Participate in rotating on-call coverage, including weekends and holidays, to diagnose issues, implement fixes, and escalating when necessary.
• Improve system reliability and observability through monitoring, logging, automated testing, documentation, and operational tooling.
• Investigate complex technical issues across backend services, data pipelines, and AI-assisted workflows, identifying root causes and implementing sustainable solutions.
• Collaborate with engineering and platform teams to deploy improvements, maintain system stability, and support evolving operational needs.
• Own engineering tasks end-to-end, from understanding user requirements and designing solutions to implementation, testing, deployment, and maintenance.

What We're Looking For

• Strong software engineering experience, particularly in Python and backend application development.
• Experience with platform engineering, site reliability engineering (SRE), or production operations.
• Experience building, deploying, debugging, and maintaining production software systems.
• Familiarity with REST APIs, backend services, asynchronous processing, and system integrations.
• Practical experience working with AI-powered applications, LLM APIs, or model-assisted development tools.
• Strong troubleshooting and debugging skills, with the ability to investigate unfamiliar systems and resolve complex production issues.
• Experience supporting production environments, including monitoring, incident response, and reliability improvements.
• Ability to independently scope technical problems, prioritize work, and deliver maintainable solutions.
• Strong communication skills and experience collaborating with technical and nontechnical stakeholders.
• Sound judgment when handling sensitive information, operational incidents, and time-sensitive technical challenges.
• Availability to participate in scheduled on-call coverage, including weekends and holidays.

Nice to Have

• Experience with asynchronous job processing, background workers, task queues, or distributed systems.
• Familiarity with cloud infrastructure and services such as AWS, GCP, or Azure.
• Experience building internal tools, workflow automation, or investigator-facing applications.
• Familiarity with observability tools, structured logging, alerting, and incident management practices.
• Experience working in cybersecurity, trust and safety, threat intelligence, or other high-stakes operational environments.

Additional Information

Employment Type: 1 year contract, 40 hours full-time

Compensation: $160,000 - $200,000

Location: Remote

Schedule: Includes rotating on-call coverage, with weekend and holiday availability required.

This role supports sensitive safety and threat investigations and may involve exposure to potentially distressing case information.