Senior Software Development Engineer III (Infrastructure)

Advanced Resource Technologies, Inc.; Washington, Northern (District of Columbia, KY); Hybrid; USD 150,000 - 210,000; Full time

Job summary

Advanced Resource Technologies, Inc. seeks a Senior Software Development Engineer III (Infrastructure) to lead backend infrastructure for GRACE on Azure, including functions, API management, container apps, and OpenAI integration.

You will own data pipelines, authentication, IaC, and CI/CD, while driving observability, security, and cost optimization. Role requires 7+ years in software engineering, strong Python skills, and hands-on Azure experience.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Software Engineering, or related field, or equivalent practical experience
• 7+ years of professional software engineering experience building and operating production systems
• Proven experience building and shipping complex products in high-velocity environments
• Strong proficiency in Python and at least one additional backend programming language
• Experience with distributed systems, APIs, data pipelines, and software design patterns
• Hands-on experience with Microsoft Azure including Azure Functions, API Management, Container Apps, and Azure OpenAI Service
• Experience with containerization, CI/CD pipelines, and infrastructure as code
• Strong understanding of OAuth2, OIDC, Azure Entra ID, or similar authentication systems
• Experience supporting production systems including incident response and postmortem analysis
• Experience building and operating MCP servers and multi-agent orchestration frameworks preferred
• Experience with LLM systems including tool-calling, RAG, multi-step reasoning, and multi-model routing preferred
• Familiarity with vector databases, embedding pipelines, and semantic search infrastructure preferred
• Strong communication, collaboration, and leadership skills
• Ability to work collaboratively with cross-functional teams
• Experience in startup, early-stage, or high-growth engineering environments preferred
• Background in regulated or government environments preferred

Responsibilities

• Own the end-to-end backend infrastructure for GRACE on Microsoft Azure, including Azure Functions, Azure API Management, Azure Container Apps, and Azure OpenAI Service
• Manage the data layer including storage, retrieval pipelines, vector databases, and document indexing supporting internal knowledge search
• Implement authentication and identity integration including ARPA-H Entra ID and application-level access control
• Maintain infrastructure as code across all environments and ensure standardized deployment practices
• Own CI/CD pipelines, deployment automation, release processes, and gradual rollout strategies
• Develop and maintain monitoring, alerting, logging, distributed tracing, SLOs, and incident response procedures
• Manage secrets, API keys, and credential rotation for external provider integrations
• Monitor and optimize token economics, LLM provider costs, and cost-per-query performance
• Implement MCP server hosting, tool registration, versioning, and lifecycle management on Azure
• Develop A2A communication patterns enabling interoperability between GRACE agents and external systems
• Design and maintain LLM orchestration and multi-model routing infrastructure across OpenAI GPT, Anthropic Claude, and Google Gemini
• Build and operate RAG pipelines including document ingestion, chunking, embedding, and semantic search
• Implement fallback, retry, circuit-breaker, and graceful degradation mechanisms for AI service dependencies
• Develop tool-calling infrastructure including execution, error handling, and observability
• Build end-to-end observability for agentic workflows including latency, throughput, token usage, and LLM quality metrics
• Implement LLM evaluation pipelines including regression monitoring, safety checks, and grounding assessments
• Define and enforce system-level SLOs for availability, response time, and reliability
• Establish coding standards, testing practices, and design review processes
• Mentor engineers and provide technical leadership across the development team
• Ensure privacy, security, and compliance across all systems and integrations

Skills

Python

Distributed systems

API design

Leadership

Education

Bachelor's or Master's in CS/CE or related

Tools

Azure Functions

Azure API Management

Azure Container Apps

Azure OpenAI Service

CI/CD

Infrastructure as Code

Job description

Senior Software Development Engineer III (Infrastructure)

Advanced Resource Technologies

Senior Software Development Engineer III (Infrastructure)

Washington, DC 20001

Advanced Resource Technologies, Inc. (ARTI) is currently recruiting for a Senior Software Development Engineer III (Infrastructure) to support the Advanced Research Projects Agency for Health (ARPA-H). This position is full-time, exempt. Start date is immediate upon selection and public trust clearance processing as required.

SUMMARY

The Senior Software Development Engineer III (Infrastructure) role would be supporting the GRACE Development Team within ARPA-H. This role focuses on backend infrastructure, agentic AI platforms, cloud architecture, and production-scale AI systems supporting GRACE, ARPA-H’s production AI assistant.

PRINCIPAL DUTIES AND RESPONSIBILITIES

• Own the end-to-end backend infrastructure for GRACE on Microsoft Azure, including Azure Functions, Azure API Management, Azure Container Apps, and Azure OpenAI Service
• Manage the data layer including storage, retrieval pipelines, vector databases, and document indexing supporting internal knowledge search
• Implement authentication and identity integration including ARPA-H Entra ID and application-level access control
• Maintain infrastructure as code across all environments and ensure standardized deployment practices
• Own CI/CD pipelines, deployment automation, release processes, and gradual rollout strategies
• Develop and maintain monitoring, alerting, logging, distributed tracing, SLOs, and incident response procedures
• Manage secrets, API keys, and credential rotation for external provider integrations
• Monitor and optimize token economics, LLM provider costs, and cost-per-query performance
• Implement MCP server hosting, tool registration, versioning, and lifecycle management on Azure
• Develop A2A communication patterns enabling interoperability between GRACE agents and external systems
• Design and maintain LLM orchestration and multi-model routing infrastructure across OpenAI GPT, Anthropic Claude, and Google Gemini
• Build and operate RAG pipelines including document ingestion, chunking, embedding, and semantic search
• Implement fallback, retry, circuit-breaker, and graceful degradation mechanisms for AI service dependencies
• Develop tool-calling infrastructure including execution, error handling, and observability
• Build end-to-end observability for agentic workflows including latency, throughput, token usage, and LLM quality metrics
• Implement LLM evaluation pipelines including regression monitoring, safety checks, and grounding assessments
• Define and enforce system-level SLOs for availability, response time, and reliability
• Establish coding standards, testing practices, and design review processes
• Mentor engineers and provide technical leadership across the development team
• Ensure privacy, security, and compliance across all systems and integrations

Job Requirements

• Bachelor’s or Master’s degree in Computer Science, Software Engineering, or related field, or equivalent practical experience
• 7+ years of professional software engineering experience building and operating production systems
• Proven experience building and shipping complex products in high-velocity environments
• Strong proficiency in Python and at least one additional backend programming language
• Experience with distributed systems, APIs, data pipelines, and software design patterns
• Hands-on experience with Microsoft Azure including Azure Functions, API Management, Container Apps, and Azure OpenAI Service
• Experience with containerization, CI/CD pipelines, and infrastructure as code
• Strong understanding of OAuth2, OIDC, Azure Entra ID, or similar authentication systems
• Experience supporting production systems including incident response and postmortem analysis
• Experience building and operating MCP servers and multi-agent orchestration frameworks preferred
• Experience with LLM systems including tool-calling, RAG, multi-step reasoning, and multi-model routing preferred
• Familiarity with vector databases, embedding pipelines, and semantic search infrastructure preferred
• Strong communication, collaboration, and leadership skills
• Ability to work collaboratively with cross-functional teams
• Experience in startup, early-stage, or high-growth engineering environments preferred
• Background in regulated or government environments preferred

Mandatory Requirement: Applicants selected for this position will be subject to a Government Security Investigation and must meet eligibility for access to classified information,Public Trust.

The above information has been designed to indicate the general nature and level of work performed by employees in this classification. It is not designed to contain or to be interpreted as a comprehensive inventory of all duties, responsibilities, and qualifications required of the employee assigned to this job.

Advanced Resource Technology, Inc. is committed to a work environment free of all forms of discrimination and member of DHS E-Verify Program.

Advanced Resource Technology, Inc. recruits and hires without regard to age, color, disability, gender, gender identity, genetic information, marital status, military status, national origin, race, religion, sexual orientation, veteran status, or any other legally protected characteristic.

Equal Opportunity Employer, including disabled and veterans.