Principal Software Engineer - Agentic SOC

vibehackers; United States; Remote; USD 215,000 - 240,000; Full time

Benefits offered by this job

Generous PTO

Parental leave

Medical, dental, vision

401(k) with company contribution

Life and Disability insurance

Stock options

Home office reimbursement

Education/professional development

Digital reimbursement

Coaching

Job summary

Huntress is seeking a Principal Software Engineer to lead the design and delivery of an Agentic Security Operations Center (SOC) built around LLM-based agents. You will own the architecture and direction of the system that investigates signals end-to-end, assembles evidence, and decides when to act autonomously or hand off to a human analyst.

You will collaborate with SOC analysts, detection engineers, product, and other engineering teams to ensure reliability, auditability, and tenant isolation

Qualifications

• 15+ years building complex software products with significant time as a technical lead on production systems.
• Hands-on experience designing, shipping, and operating LLM-based agents in production (tool use, context management, structured output, guardrails, evaluation) and an understanding of failure modes.
• Experience building evaluation frameworks for non-deterministic systems (labeled datasets, offline evals, regression gates).
• Security mindset with experience handling untrusted input, multi-tenant isolation, and auditable trails.
• Experience building systems that make automated decisions with real consequences and sound judgment about what to automate.
• Deep backend expertise in distributed systems, queues, durable workflows, and concurrency; designing for throughput and correctness under load.
• Strong skills in one or more backend languages and ability to learn new ones quickly; primary stack is Ruby on Rails and fluency is expected.
• Experience with public cloud environments (AWS, Azure, or similar) and data stores such as Postgres and Redis.
• Familiarity with AI coding tools (example: Claude Code).
• BS or MS in Computer Science/Engineering or equivalent experience.

Responsibilities

• Own the architecture of agentic investigations: signal ingestion, investigation state management, tool surfaces for agents, and final investigation artifacts (verdict, timeline, scope, evidence).
• Design and build LLM-powered agents that replicate strong analyst behavior and make appropriate decisions about escalation and handoff.
• Build LLM-based pre-processing to summarize, correlate, and surface what matters from analyst tools and telemetry.
• Embed a security mindset in the agent design: trust boundaries, tenant isolation, attack resistance, and full audit trails.
• Define and own evaluation: labeled datasets, offline evaluation, regression gates, and metrics to measure non-deterministic system quality over time.
• Decide, based on data, what the system can act on autonomously and what requires human review; evolve that decisioning responsibly.
• Engineer for production: reliability, latency, cost at scale, and robustness to model failures or changes.
• Prototype and derisk uncertain approaches, then hand off production-ready designs to the team.
• Represent the team technically across SOC leadership, product, other engineering teams, and executives; drive alignment and protect team scope.
• Mentor engineers, perform thoughtful code review, and raise both the team’s baseline and ceiling through patterns and tooling.

Skills

System Architecture

System Design

Distributed Systems

Durable Workflows

Queues

Concurrency

Reliability Engineering

Performance and Cost Optimization

LLM/Agent Design

Evaluation of Non-deterministic Sys

Security Mindset

Multi-tenant Isolation

Auditability

Prototyping

Technical Leadership

Mentorship

Cross-functional Communication

Data-driven Decision Making

Code Review

Education

BS or MS in Computer Science/Engineering or equivalent experience

Tools

Ruby on Rails

Postgres

Redis

AWS

Azure

Claude Code

Job description

Principal Software Engineer - Agentic SOC

Heavy use of LLMs and AI coding tools to build agentic systems; role centers on building and evaluating LLM-based agents and rapid prototyping with AI.

About the Role

Lead the design and implementation of an Agentic SOC that uses LLM-based agents to investigate security signals end-to-end, augmenting SOC analysts by automating high-volume, repetitive work while preserving security, auditability, and human handoffs. Own system architecture, reliability, evaluation, and technical leadership for production-grade agent behavior and decision-making.

Job Description

Role

Huntress is hiring a Principal Software Engineer to lead the design and delivery of an Agentic Security Operations Center (SOC). The role owns the architecture and technical direction of an LLM-based agent system that investigates signals end-to-end, assembles evidence, and decides when to act autonomously versus hand off to a human analyst. You will work closely with SOC analysts, detection engineers, product, and other engineering teams and drive production-grade reliability, auditability, and tenant isolation.

Key Responsibilities

• Own the architecture of agentic investigations: signal ingestion, investigation state management, tool surfaces for agents, and final investigation artifacts (verdict, timeline, scope, evidence).
• Design and build LLM-powered agents that replicate strong analyst behavior and make appropriate decisions about escalation and handoff.
• Build LLM-based pre-processing to summarize, correlate, and surface what matters from analyst tools and telemetry.
• Embed a security mindset in the agent design: trust boundaries, tenant isolation, attack resistance, and full audit trails.
• Define and own evaluation: labeled datasets, offline evaluation, regression gates, and metrics to measure non-deterministic system quality over time.
• Decide, based on data, what the system can act on autonomously and what requires human review; evolve that decisioning responsibly.
• Engineer for production: reliability, latency, cost at scale, and robustness to model failures or changes.
• Prototype and derisk uncertain approaches, then hand off production-ready designs to the team.
• Represent the team technically across SOC leadership, product, other engineering teams, and executives; drive alignment and protect team scope.
• Mentor engineers, perform thoughtful code review, and raise both the team’s baseline and ceiling through patterns and tooling.

Requirements

• 15+ years building complex software products with significant time as a technical lead on production systems.
• Hands‑on experience designing, shipping, and operating LLM‑based agents in production (tool use, context management, structured output, guardrails, evaluation) and an understanding of failure modes.
• Experience building evaluation frameworks for non-deterministic systems (labeled datasets, offline evals, regression gates).
• Security mindset with experience handling untrusted input, multi‑tenant isolation, and auditable trails.
• Experience building systems that make automated decisions with real consequences and sound judgment about what to automate.
• Deep backend expertise in distributed systems, queues, durable workflows, and concurrency; designing for throughput and correctness under load.
• Strong skills in one or more backend languages and ability to learn new ones quickly; primary stack is Ruby on Rails and fluency is expected.
• Experience with public cloud environments (AWS, Azure, or similar) and data stores such as Postgres and Redis.
• Familiarity with AI coding tools (example: Claude Code).
• BS or MS in Computer Science/Engineering or equivalent experience.

Bonus Qualifications

• Experience in SOCs, incident response, threat hunting, or detection engineering.
• Familiarity with endpoint and identity telemetry and attacker tradecraft.
• Experience adversarially testing ML/LLM systems.
• Additional experience with Ruby on Rails.

Compensation & Location

• Base: $215,000 to $240,000 (plus bonus and equity).
• Location: Remote (U.S. only).

Huntress provides 100% remote work, generous PTO (vacation, sick, holidays), 12 weeks paid parental leave, comprehensive medical/dental/vision plans, 401(k) with 5% contribution regardless of employee contribution, life and disability insurance, stock options for full-time employees, $500 one-time home office reimbursement, annual education/professional development allowance, $75/month digital reimbursement, and access to coaching (BetterUp).

Ruby on Rails Postgres Redis AWS Azure LLMs Claude Code

Skills

System Architecture System Design Distributed Systems Durable Workflows Queues Concurrency Reliability Engineering Performance and Cost Optimization LLM/Agent Design Evaluation of Non-deterministic Systems Security Mindset Multi-tenant Isolation Auditability Prototyping Technical Leadership Mentorship Cross-functional Communication Data-driven Decision Making Code Review

Experience Level

USD 215,000 - 240,000/year

Employment Type

{Full-Time}

• Generous paid time off (vacation, sick, holidays)
• 12 weeks paid parental leave
• Medical, dental, and vision benefits
• 401(k) with 5% company contribution
• Life and Disability insurance
• Stock options for full-time employees
• One-time $500 home office reimbursement
• Annual education/professional development allowance