4769-Principal Software Architect

Innovaccer Analytics; United States; On-site; USD 190,000 - 250,000; Full time

Benefits offered by this job

Generous PTO

Parental leave

Job summary

Innovaccer is seeking a Principal Engineer to own the architecture for our AI-driven contact center platform. You will design real-time, low-latency pipelines and an orchestration layer for AI agents across voice, chat, and SMS, with a focus on safety, observability, and scalable systems.

You will mentor engineers, drive architecture decisions across multiple services, and collaborate with product teams to scope ambitious initiatives while maintaining strict security and compliance for

Qualifications

• 8+ years of backend engineering with ownership of a multi-service system.

Responsibilities

• Own architecture for AI voice agent platform and next-best-action guidance.
• Design real-time, low-latency pipelines (sub-second) for transcription and inference.
• Expose agent workflows as reusable actions for the broader system.
• Define identity, context retrieval, and escalation contracts with audit trails.
• Drive reliability metrics and set SLOs for agent services.
• Lead technical design reviews across backend services.
• Mentor engineers on distributed-systems and AI-agent design.
• Collaborate with product to scope initiatives and plan phased work.
• Apply security and compliance practices for healthcare data.

Skills

Backend engineering

Python async

FastAPI/Django

Distributed systems

Kafka

MongoDB

PostgreSQL

LLM frameworks

Architecture leadership

Docker CI/CD

Git workflows

Tools

Docker

CI/CD

Observability

Job description

About the Role

We at Innovaccer are looking for a Principal Engineer to build and own the most amazing product experience for our contact center. You'll get to work with other engineers to build delightful feature experiences to understand and solve our customers' pain points. Our backend platform powers a healthcare contact center and AI agent product used by health systems and clinics, running as a set of async, containerized services across multiple live customer deployments, with a growing AI agent stack spanning voice, chat, and SMS. Beyond core ticketing and scheduling workflows, the platform is investing in two strategic capabilities: standalone AI voice/chat/SMS agents that can operate with minimal platform dependency, and an in-call "next best action" layer that turns today's reactive AI assistant into a proactive one.

A Day in the Life

Own the architecture for one or more of: the AI voice agent platform (conversational AI pathway orchestration, call routing, inbound/outbound flows), next-best-action AI assistant guidance, or standalone AI agents that operate across voice, chat, and SMS with minimal dependency on the core platform. Design real-time, low-latency systems for in-call intent detection and next-best-action suggestion (sub-second latency budgets across transcription, speaker separation, and inference). Design an orchestration layer that exposes AI agent workflows as reusable, callable actions consumable by other systems (e.g. a broader automation/journey engine), including output contracts and versioning. Define patient/customer identification, context retrieval, and escalation contracts for agents operating with minimal identifiers, ensuring safe failure and handoff to a human with a full audit trail. Drive the AI agent evaluation and reliability framework (task completion rate, containment rate, transfer rate, conversion/booking rate, webhook success rate, P50/P95/P99 latency) and set SLOs for agent-facing services. Lead technical design reviews (high- and low-level designs) across backend services (core ticketing, workqueue/case management, scheduling, transcription and speech processing) and drive architectural consistency across them. Design and build backend services in Python (FastAPI async, some Django), model data across MongoDB and PostgreSQL, and build Kafka-based event streams for cross-service and agent-to-core communication. Work hands-on with LLM/AI agent tooling (e.g. LangChain, pydantic-ai, MLflow or equivalents) for scheduling agents, in-call recommendations, and automated documentation. Set the bar for testing, observability, and production triage practices across the team; mentor senior and mid-level engineers on distributed-systems and AI-agent design problems. Partner with product on scoping and sequencing ambiguous initiatives, translating them into phased technical plans. Apply security and compliance practices relevant to healthcare data (PHI handling, audit logging, OAuth/JWT auth, "minimum necessary" data scoping for agent context retrieval).

What You Need

• 8+ years of backend engineering experience, including demonstrated technical ownership of a complex, multi-service system (not just feature delivery)
• Strong proficiency in Python, including async programming (asyncio), with production experience in FastAPI (or Django)
• Experience designing distributed, multi-service architectures: event-driven systems (Kafka or similar), service-to-service auth, webhook-based integrations, and idempotency/retry design
• Experience with MongoDB or another NoSQL database, and working knowledge of relational databases (PostgreSQL)
• Direct experience building or operating conversational AI / voice agent systems in production (telephony, IVR, or conversational voice platforms), including latency-sensitive real-time pipelines (speech-to-text, speaker separation, text-to-speech)
• Experience designing systems that make real-time decisions from streaming context (e.g. intent detection, recommendation surfacing, next-best-action logic) with confidence thresholds and human-in-the-loop escalation
• Experience with LLM/agent frameworks and a working understanding of agent evaluation and reliability practices (task success metrics, failure taxonomies, monitoring)
• Track record of driving architecture decisions across teams, writing and reviewing technical design docs, and mentoring other engineers
• Comfortable with Docker/containerized deployments, CI/CD, and observability tooling for diagnosing production issues across a service mesh
• Proficient with Git and collaborative workflows

What We Offer

• Generous Paid Time Off: Recharge and relax with 20 days of fixed time off per year, in addition to company holidays-because we believe work-life balance fuels performance.
• Best-in-Class Parental Leave: Spend quality time with your growing family. We offer one of the industry's most generous parental leave policies to support you.