Senior AI Infrastructure Engineer – Azure & LLM Systems

Advanced Resource Technologies, Inc.; Washington, Northern (District of Columbia, KY); Hybrid; USD 150,000 - 210,000; Full time

Job summary

Advanced Resource Technologies, Inc. seeks a Senior Software Development Engineer III (Infrastructure) to lead backend infrastructure for GRACE on Azure, including functions, API management, container apps, and OpenAI integration.

You will own data pipelines, authentication, IaC, and CI/CD, while driving observability, security, and cost optimization. Role requires 7+ years in software engineering, strong Python skills, and hands-on Azure experience.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Software Engineering, or related field, or equivalent practical experience
• 7+ years of professional software engineering experience building and operating production systems
• Proven experience building and shipping complex products in high-velocity environments
• Strong proficiency in Python and at least one additional backend programming language
• Experience with distributed systems, APIs, data pipelines, and software design patterns
• Hands-on experience with Microsoft Azure including Azure Functions, API Management, Container Apps, and Azure OpenAI Service
• Experience with containerization, CI/CD pipelines, and infrastructure as code
• Strong understanding of OAuth2, OIDC, Azure Entra ID, or similar authentication systems
• Experience supporting production systems including incident response and postmortem analysis
• Experience building and operating MCP servers and multi-agent orchestration frameworks preferred
• Experience with LLM systems including tool-calling, RAG, multi-step reasoning, and multi-model routing preferred
• Familiarity with vector databases, embedding pipelines, and semantic search infrastructure preferred
• Strong communication, collaboration, and leadership skills
• Ability to work collaboratively with cross-functional teams
• Experience in startup, early-stage, or high-growth engineering environments preferred
• Background in regulated or government environments preferred

Responsibilities

• Own the end-to-end backend infrastructure for GRACE on Microsoft Azure, including Azure Functions, Azure API Management, Azure Container Apps, and Azure OpenAI Service
• Manage the data layer including storage, retrieval pipelines, vector databases, and document indexing supporting internal knowledge search
• Implement authentication and identity integration including ARPA-H Entra ID and application-level access control
• Maintain infrastructure as code across all environments and ensure standardized deployment practices
• Own CI/CD pipelines, deployment automation, release processes, and gradual rollout strategies
• Develop and maintain monitoring, alerting, logging, distributed tracing, SLOs, and incident response procedures
• Manage secrets, API keys, and credential rotation for external provider integrations
• Monitor and optimize token economics, LLM provider costs, and cost-per-query performance
• Implement MCP server hosting, tool registration, versioning, and lifecycle management on Azure
• Develop A2A communication patterns enabling interoperability between GRACE agents and external systems
• Design and maintain LLM orchestration and multi-model routing infrastructure across OpenAI GPT, Anthropic Claude, and Google Gemini
• Build and operate RAG pipelines including document ingestion, chunking, embedding, and semantic search
• Implement fallback, retry, circuit-breaker, and graceful degradation mechanisms for AI service dependencies
• Develop tool-calling infrastructure including execution, error handling, and observability
• Build end-to-end observability for agentic workflows including latency, throughput, token usage, and LLM quality metrics
• Implement LLM evaluation pipelines including regression monitoring, safety checks, and grounding assessments
• Define and enforce system-level SLOs for availability, response time, and reliability
• Establish coding standards, testing practices, and design review processes
• Mentor engineers and provide technical leadership across the development team
• Ensure privacy, security, and compliance across all systems and integrations

Skills

Python

Distributed systems

API design

Leadership

Education

Bachelor's or Master's in CS/CE or related

Tools

Azure Functions

Azure API Management

Azure Container Apps

Azure OpenAI Service

CI/CD

Infrastructure as Code

Job description

Advanced Resource Technologies, Inc. seeks a Senior Software Development Engineer III (Infrastructure) to lead backend infrastructure for GRACE on Azure, including functions, API management, container apps, and OpenAI integration.

You will own data pipelines, authentication, IaC, and CI/CD, while driving observability, security, and cost optimization. Role requires 7+ years in software engineering, strong Python skills, and hands-on Azure experience.