Principal Software Engineer - Agentic SOC (Remote)

vibehackers; United States; Remote; USD 215,000 - 240,000; Full time

Benefits offered by this job

Generous PTO

Parental leave

Medical, dental, vision

401(k) with company contribution

Life and Disability insurance

Stock options

Home office reimbursement

Education/professional development

Digital reimbursement

Coaching

Job summary

Huntress is seeking a Principal Software Engineer to lead the design and delivery of an Agentic Security Operations Center (SOC) built around LLM-based agents. You will own the architecture and direction of the system that investigates signals end-to-end, assembles evidence, and decides when to act autonomously or hand off to a human analyst.

You will collaborate with SOC analysts, detection engineers, product, and other engineering teams to ensure reliability, auditability, and tenant isolation

Qualifications

• 15+ years building complex software products with significant time as a technical lead on production systems.
• Hands-on experience designing, shipping, and operating LLM-based agents in production (tool use, context management, structured output, guardrails, evaluation) and an understanding of failure modes.
• Experience building evaluation frameworks for non-deterministic systems (labeled datasets, offline evals, regression gates).
• Security mindset with experience handling untrusted input, multi-tenant isolation, and auditable trails.
• Experience building systems that make automated decisions with real consequences and sound judgment about what to automate.
• Deep backend expertise in distributed systems, queues, durable workflows, and concurrency; designing for throughput and correctness under load.
• Strong skills in one or more backend languages and ability to learn new ones quickly; primary stack is Ruby on Rails and fluency is expected.
• Experience with public cloud environments (AWS, Azure, or similar) and data stores such as Postgres and Redis.
• Familiarity with AI coding tools (example: Claude Code).
• BS or MS in Computer Science/Engineering or equivalent experience.

Responsibilities

• Own the architecture of agentic investigations: signal ingestion, investigation state management, tool surfaces for agents, and final investigation artifacts (verdict, timeline, scope, evidence).
• Design and build LLM-powered agents that replicate strong analyst behavior and make appropriate decisions about escalation and handoff.
• Build LLM-based pre-processing to summarize, correlate, and surface what matters from analyst tools and telemetry.
• Embed a security mindset in the agent design: trust boundaries, tenant isolation, attack resistance, and full audit trails.
• Define and own evaluation: labeled datasets, offline evaluation, regression gates, and metrics to measure non-deterministic system quality over time.
• Decide, based on data, what the system can act on autonomously and what requires human review; evolve that decisioning responsibly.
• Engineer for production: reliability, latency, cost at scale, and robustness to model failures or changes.
• Prototype and derisk uncertain approaches, then hand off production-ready designs to the team.
• Represent the team technically across SOC leadership, product, other engineering teams, and executives; drive alignment and protect team scope.
• Mentor engineers, perform thoughtful code review, and raise both the team’s baseline and ceiling through patterns and tooling.

Skills

System Architecture

System Design

Distributed Systems

Durable Workflows

Queues

Concurrency

Reliability Engineering

Performance and Cost Optimization

LLM/Agent Design

Evaluation of Non-deterministic Sys

Security Mindset

Multi-tenant Isolation

Auditability

Prototyping

Technical Leadership

Mentorship

Cross-functional Communication

Data-driven Decision Making

Code Review

Education

BS or MS in Computer Science/Engineering or equivalent experience

Tools

Ruby on Rails

Postgres

Redis

AWS

Azure

Claude Code

Job description

Huntress is seeking a Principal Software Engineer to lead the design and delivery of an Agentic Security Operations Center (SOC) built around LLM-based agents. You will own the architecture and direction of the system that investigates signals end-to-end, assembles evidence, and decides when to act autonomously or hand off to a human analyst.

You will collaborate with SOC analysts, detection engineers, product, and other engineering teams to ensure reliability, auditability, and tenant isolation