4769-Principal Software Architect

Innovaccer Inc.; United States; Remote; USD 180,000 - 260,000; Full time

Benefits offered by this job

Paid Time Off

Parental Leave

Recognition & Rewards

Insurance Coverage

Job summary

Innovaccer Inc. is seeking a Principal Engineer to own architecture for our AI-enabled contact center backend platform.

You will work with a team of engineers to build real-time, low-latency systems and expose reusable AI agent workflows across services, ensuring safe handoffs to humans with full audit trails. The role requires deep expertise in Python, async programming, distributed systems, and experience with MongoDB and PostgreSQL.

Qualifications

• 8+ years of backend engineering with ownership of a multi-service system.

Responsibilities

• Own the architecture for one or more AI-enabled backend platforms and multi-service systems.
• Design real-time, low-latency pipelines for AI features (voice, chat, SMS).
• Build orchestration layers exposing agent workflows and versioned outputs.
• Define identification, context retrieval, and escalation contracts with audit trails.
• Drive agent evaluation and reliability metrics (task completion, latency, webhook success).
• Lead architecture reviews across backend services (ticketing, scheduling, transcription).
• Develop backend services in Python (FastAPI/Django), model data in MongoDB and PostgreSQL, and Kafka streams.
• Work with LLM tooling (LangChain, pydantic-ai, MLflow) for agent scheduling and recommendations.
• Set testing, observability, and production triage standards; mentor engineers on distributed systems and AI design.
• Collaborate with product to scope initiatives and translate into phased technical plans.
• Apply healthcare data security and compliance practices (PHI, OAuth/JWT, minimum necessary data).

Skills

Backend engineering

Python

Async programming

Distributed systems

MongoDB

PostgreSQL

LLM/agent frameworks

Docker

CI/CD

Git

Observability

Tools

FastAPI

Django

Kafka

Job description

About the Role

We at Innovaccer are looking for a Principal Engineer to build and own the most amazing product experience for our contact center. You’ll get to work with other engineers to build delightful feature experiences to understand and solve our customers’ pain points. Our backend platform powers a healthcare contact center and AI agent product used by health systems and clinics, running as a set of async, containerized services across multiple live customer deployments, with a growing AI agent stack spanning voice, chat, and SMS. Beyond core ticketing and scheduling workflows, the platform is investing in two strategic capabilities: standalone AI voice/chat/SMS agents that can operate with minimal platform dependency, and an in-call "next best action" layer that turns today's reactive AI assistant into a proactive one.

A Day in the Life

• Own the architecture for one or more of: the AI voice agent platform (conversational AI pathway orchestration, call routing, inbound/outbound flows), next-best-action AI assistant guidance, or standalone AI agents that operate across voice, chat, and SMS with minimal dependency on the core platform UI
• Design real-time, low-latency systems for in-call intent detection and next-best-action suggestion (sub-second latency budgets across transcription, speaker separation, and inference)
• Design an orchestration layer that exposes AI agent workflows as reusable, callable actions consumable by other systems (e.g. a broader automation/journey engine), including output contracts and versioning
• Define patient/customer identification, context retrieval, and escalation contracts for agents operating with minimal identifiers, ensuring safe failure and handoff to a human with a full audit trail
• Drive the AI agent evaluation and reliability framework (task completion rate, containment rate, transfer rate, conversion/booking rate, webhook success rate, P50/P95/P99 latency) and set SLOs for agent-facing services
• Lead technical design reviews (high- and low-level designs) across backend services (core ticketing, workqueue/case management, scheduling, transcription and speech processing) and drive architectural consistency across them
• Design and build backend services in Python (FastAPI async, some Django), model data across MongoDB and PostgreSQL, and build Kafka-based event streams for cross-service and agent-to-core communication
• Work hands-on with LLM/AI agent tooling (e.g. LangChain, pydantic-ai, MLflow or equivalents) for scheduling agents, in-call recommendations, and automated documentation
• Set the bar for testing, observability, and production triage practices across the team; mentor senior and mid-level engineers on distributed-systems and AI-agent design problems
• Partner with product on scoping and sequencing ambiguous initiatives, translating them into phased technical plans
• Apply security and compliance practices relevant to healthcare data (PHI handling, audit logging, OAuth/JWT auth, "minimum necessary" data scoping for agent context retrieval)

What You Need

• 8+ years of backend engineering experience, including demonstrated technical ownership of a complex, multi-service system (not just feature delivery)
• Strong proficiency in Python, including async programming (asyncio), with production experience in FastAPI (or Django)
• Experience designing distributed, multi-service architectures: event-driven systems (Kafka or similar), service-to-service auth, webhook-based integrations, and idempotency/retry design
• Experience with MongoDB or another NoSQL database, and working knowledge of relational databases (PostgreSQL)
• Direct experience building or operating conversational AI / voice agent systems in production (telephony, IVR, or conversational voice platforms), including latency-sensitive real-time pipelines (speech-to-text, speaker separation, text-to-speech)
• Experience designing systems that make real-time decisions from streaming context (e.g. intent detection, recommendation surfacing, next-best-action logic) with confidence thresholds and human-in-the-loop escalation
• Experience with LLM/agent frameworks and a working understanding of agent evaluation and reliability practices (task success metrics, failure taxonomies, monitoring)
• Track record of driving architecture decisions across teams, writing and reviewing technical design docs, and mentoring other engineers
• Comfortable with Docker/containerized deployments, CI/CD, and observability tooling for diagnosing production issues across a service mesh
• Proficient with Git and collaborative workflows

We offer competitive benefits to set you up for success in and outside of work.

Here’s What We Offer

• Generous Paid Time Off: Recharge and relax with 20 days of fixed time off per year, in addition to company holidays—because we believe work-life balance fuels performance.
• Best-in-Class Parental Leave: Spend quality time with your growing family. We offer one of the industry’s most generous parental leave policies to support you during life’s most important moments.
• Recognition & Rewards: We celebrate wins—big and small. Get rewarded with monetary incentives and company-wide recognition for your impact and dedication. Your hard work won’t go unnoticed.
• Comprehensive Insurance Coverage: Stay covered with medical, dental, and vision insurance, plus 100% company-paid short- and long-term disability and basic life insurance. Optional perks include discounted legal aid and pet insurance.

Innovaccer Inc. is an equal opportunity employer. We celebrate diversity and are committed to fostering an inclusive workplace where all employees feel valued and empowered regardless of any characteristic protected by federal, state or local law including, without limitation, race, color, religion, gender, gender identity or expression, sexual orientation, national origin, genetics, medical condition, disability, age, marital status, or veteran status. Innovaccer Inc. participates in the E-Verify program to confirm employment eligibility of all newly hired employees based out of the U.S. and employed by Innovaccer Inc.

For any additional information, please visit the below websites:

E-Verify

Right to Work (English)

Right to Work (Spanish)

About Innovaccer

Innovaccer activates the flow of healthcare data, empowering providers, payers, and government organizations to deliver intelligent and connected experiences that advance health outcomes. The Healthcare Intelligence Cloud equips every stakeholder in the patient journey to turn fragmented data into proactive, coordinated actions that elevate the quality of care and drive operational performance. Leading healthcare organizations like CommonSpirit Health, Atlantic Health, and Banner Health trust Innovaccer to integrate a system of intelligence into their existing infrastructure, extending the human touch in healthcare. For more information, visit www.innovaccer.com.

Check us out on YouTube, Glassdoor, LinkedIn, Instagram, and the Web.