Senior AI Engineer (Agentic Systems)

Pantheon Data; Reston (VA); Hybrid; USD 120,000 - 220,000; Full time

Benefits offered by this job

SmartBenefits program

Tuition assistance

Flexible work options

Job summary

Pantheon Data in Reston, VA is seeking a Senior AI Engineer (Agentic Systems) to design, build, and operate the agentic framework at the core of its AI platform in AWS GovCloud. You will orchestrate agents, route requests, and manage the capability registry to deliver secure, auditable, and scalable workflows for workloads involving CUI, PII, and PHI.

This is a hands-on leadership role requiring production-ready systems experience and FedRAMP/DoD context.

Qualifications

• Bachelor's degree in Computer Science, Engineering, or a related field or equivalent experience.
• 8+ years of professional software engineering experience, including 2+ years building production LLM-backed systems.
• Experience designing and shipping production agentic systems with multi-step/workflows.
• Hands-on experience with LLM tool calling and at least one agent framework/SDK.
• Strong software architecture skills: plugin/registry designs, versioning and extensibility.
• Experience with distributed/asynchronous systems, queues, retries, idempotency, and failure handling.
• Strong Python engineering skills with tested code and API/service design.
• Knowledge of AI security risks and mitigations, including prompt injection and least-privilege access.

Responsibilities

• Design and build the agent orchestration layer for multi-agent workflows.
• Develop routing that maps requests to agents, tools, and models with observable decisions.
• Own the capability registry and runtime assembly of agents from registered capabilities.
• Define interfaces and contracts for tool integrations and MCP servers.
• Design agent state, memory, and reliability features with retries and graceful degradation.
• Build evaluation harnesses and end-to-end traceability for debugging and cost control.
• Enforce security and auditability with least-privilege access and audit trails.
• Provide technical leadership and mentorship across the team.

Skills

Python

LLM tool calling

Distributed systems

Cloud security awareness

System orchestration

Observability

Education

Bachelor's degree in Computer Science or Engineering

Tools

LangGraph

Strands Agents

Claude Agent SDK

Semantic Kernel

AWS Bedrock

Lambda

Step Functions

ECS / EKS

SQS

DynamoDB

CloudWatch

Terraform

GitHub Actions

GitLab CI

Job description

Company Overview

Pantheon Data (a Kenific Holding company) is a private, small business based in the Washington, DC, area. Pantheon Data was founded in 2011, initially providing acquisition and supply chain management services to the US Coast Guard. Our service offerings have grown in the past ten years, including infrastructure resiliency, contact center operations, information technology, software engineering, program management, strategic communications, engineering, and cybersecurity. We have also grown our customer base to include commercial clients. The company has used this experience to expand our service offerings to other agencies within the Department of Homeland Security (DHS), the Department of Defense (DoD), and other Federal Civilian Agencies.

Position Overview

Pantheon Data is seeking a Senior AI Engineer (Agentic Systems) to design, build, and operate the agentic framework at the core of our AI platform in AWS GovCloud, serving workloads that process CUI, PII, and PHI. You will own the framework that routes requests to the right agents, composes new agents at runtime from a registry of capabilities, and coordinates their tools, state, and handoffs, so that these systems behave predictably, securely, and auditably under FedRAMP High and DoD IL4/IL5. This role owns the agent runtime: orchestration, routing, the capability registry, and the contracts that capabilities plug into. Capabilities such as retrieval, document extraction, and application integrations are built by machine learning and software engineers and exposed through the registry; you define how they are described, discovered, composed, and governed. You will partner closely with the Data Engineer, who owns the pipelines and data stores agents draw on; the Cloud Security Engineer, who owns cloud security controls and ATO evidence; and the DevSecOps and Cloud Operations Engineers, who own delivery and infrastructure. This is a systems engineering role, not a prompt-only or prototype role: successful candidates can walk through production agentic systems they have personally built, including the orchestration design, how routing decisions were made and evaluated, how failures were contained, and what they would change.

Responsibilities

• Agent Orchestration: Design and build the orchestration layer that plans, dispatches, and coordinates work across agents, including multi-step and multi-agent workflows, handoffs, parallel execution, cancellation, and human-in-the-loop checkpoints.
• Routing: Build the routing that maps requests and intermediate steps to the right agents, tools, and models, combining deterministic rules, classifiers, and LLM-based routing where each is appropriate; measure routing accuracy and cost, and make routing decisions observable and explainable.
• Capability Registry and Dynamic Agents: Own the capability registry, including schemas for describing capabilities, versioning, discovery, and access policy, and the mechanisms that assemble agents at runtime from registered capabilities, tools, and instructions.
• Tool and Integration Contracts: Define the interfaces capabilities plug into, including tool definitions, Model Context Protocol (MCP) servers, input and output schemas, and error semantics, and help capability owners onboard to them.
• State, Memory, and Reliability: Design agent state, session memory, and context management; build retries, timeouts, idempotency, loop and budget limits, and graceful degradation so agent failures are contained and recoverable.
• Evaluation and Observability: Build evaluation harnesses for agent behavior, including routing accuracy, task completion, tool-call correctness, and regression across model and prompt versions, and trace every agent run end to end for debugging, cost, and latency.
• Agent Security and Auditability: Enforce least-privilege tool access per agent and per request, contain prompt injection across tool and agent boundaries, and record a complete audit trail of agent decisions and actions; work with the Cloud Security Engineer to map these controls to NIST SP 800-53 and support ATO.
• Technical Leadership: Set the architecture and standards for agentic development across the team, review designs and code, and mentor engineers building capabilities on the framework.

Required Skills and Experience

• Bachelor's degree in Computer Science, Engineering, or a related technical field from an accredited university, or equivalent professional experience.
• 8+ years of professional software engineering experience, including 2+ years building production LLM-backed systems.
• Demonstrated experience designing and shipping a production agentic system: multi-step or multi-agent orchestration with tool use, beyond single-prompt chat applications or demos.
• Hands‑on experience with LLM tool calling, structured outputs, and at least one agent framework or SDK, such as LangGraph, Strands Agents, the Claude Agent SDK, Semantic Kernel, or a custom framework you built.
• Strong software architecture skills: plugin or registry-based designs, interface and schema design, versioning, and extensibility.
• Experience with distributed and asynchronous systems: queues, event-driven workflows, concurrency, retries, idempotency, and failure handling.
• Experience evaluating nondeterministic systems, including building test sets, automated evaluations, and regression checks for LLM behavior.
• Strong Python engineering skills: readable, maintainable, tested code; debugging; packaging; and API and service design.
• Working knowledge of AI security risks such as prompt injection, excessive tool permissions, and data exfiltration, and of controls that mitigate them.
• Experience with AWS services relevant to this work, such as Amazon Bedrock, Lambda, Step Functions, ECS or EKS, SQS, DynamoDB, and CloudWatch.
• Experience with Git-based workflows, code review, CI/CD, and team-based software delivery.
• Clear written and verbal communication, including explaining agent behavior, tradeoffs, and failure modes to technical and non-technical stakeholders.
• Ability to work effectively remotely in cross-functional teams.
• Ability to meet deadlines and produce quality work.
• Proficient in Microsoft Suite software including Outlook, Word, Excel, SharePoint, and PowerPoint.

Preferred Skills and Experience

• Experience building or operating agentic systems in AWS GovCloud or other environments supporting FedRAMP High or DoD IL4/IL5 workloads.
• Experience implementing or integrating MCP servers and clients, or agent-to-agent protocols, in production.
• Experience with Amazon Bedrock beyond basic inference: Agents, Guardrails, Knowledge Bases, AgentCore, provisioned throughput, and cross-model routing.
• Experience building model routing or cascading strategies that balance quality, latency, and cost across multiple models.
• Experience with agent observability and tracing tools such as OpenTelemetry, Langfuse, LangSmith, or Arize Phoenix.
• Experience integrating retrieval, document processing, or other ML capabilities as tools within an agentic framework.
• Familiarity with federal compliance frameworks and their engineering implications: NIST SP 800-53, CUI handling, audit logging, and least-privilege access in regulated environments.
• Experience with infrastructure as code (Terraform), containerized deployment, and GitLab CI or GitHub Actions in regulated environments.
• Advanced degree in a technical discipline, or demonstrated contribution to team technical direction and mentorship of other engineers.

Clearance Requirements

Applicants selected will be subject to a security investigation and may need to meet eligibility requirements. Candidates must have the ability to obtain and maintain a Secret clearance; an active Secret clearance is preferred.

Work Location

Reston, VA - Remote

• Our company prioritizes the benefits of flexibility and collaboration, whether that happens in person or remotely.
• If the position is remote or hybrid, you may periodically work from a Pantheon Data office location or client site.
• If this position is assigned to a Pantheon Data office location or client site, you'll work with colleagues and clients in person, as needed for specific client requirements.

Interview Requirement

Candidates who are local to the area should be prepared to participate in an in-person interview as part of the selection process. Candidates outside the local area may be considered for a virtual interview.

Compensation

The salary range for this position is $120,000 - $220,000. This is not, however, a guarantee of compensation or salary. Rather, salary will be set based on experience, geographic location and possibly contractual requirements and could fall outside of this range.

Benefits Overview

We are always looking for good people! Pantheon Data is committed to providing its employees with competitive salaries and benefits in order to increase employee satisfaction and productivity. In addition to our benefits, we also offer SmartBenefits through the Washington Metro Area Transportation Authority, where you specify an amount of your pre-tax wages be paid directly to your SmarTrip account. In some cases, tuition assistance may be available for continuing education expenses and certifications related to their position. Additional details may be found at https://pantheon-data.com/careers/

Pantheon Data Important Information

All qualified applicants will be considered for employment without regard to disability, status as a protected veteran, or any other status protected by applicable federal, state, local, or international law.

As part of the application process, you are expected to be on camera during interviews and assessments. We reserve the right to take your picture to verify your identity and prevent fraud.

If you require reasonable accommodation in completing this application, interviewing, completing any pre-employment testing, or otherwise participating in the employee selection process, please direct your inquiries to our Talent Team at [e-mail] or by phone [phone].

This company uses E-Verify to confirm each employee's work authorization. For more information, click here E-Verify Participation Poster