Principal Real-Time AI Systems Engineer

Hispanic Alliance for Career Enhancement; Montpelier (VT); On-site; USD 144,200 - 288,400; Full time

Benefits offered by this job

Medical, dental, vision coverage

Paid time off

Retirement savings options

Job summary

CVS Health is actively hiring a Principal Software Engineer to lead the technical direction of a real-time clinical decision-support platform on Google Cloud. You will personally build critical, performance-sensitive components while mentoring engineers across UI, backend, and ML disciplines.

The role requires 10+ years in software engineering, hands-on staff/leadership, and extensive experience with Python, distributed systems, and GCP-based ML workflows.

Qualifications

• 10+ years of software engineering experience with a strong record of delivering production systems at scale.
• 5+ years as a hands-on Staff or Principal-level engineer (actively coding, not removed from implementation)
• Expert proficiency in Python including performance tuning, concurrency, and system optimization
• 5+ years building real-time distributed systems (event-driven, streaming, Pub-Sub) with clear latency and reliability requirements
• 5+ years working in a major public cloud (GCP preferred: Vertex AI, GKE, Cloud Run, Pub/Sub, BigQuery, IAM)
• Deep experience with microservices architecture, API design (REST), and resilient systems (retries, backpressure, tracing)
• 3+ years building and operating production ML systems (MLOps, monitoring, retraining, deployment pipelines)
• 3+ years developing LLM-powered applications in production, including prompt design, RAG/vector search, evaluation frameworks, and performance optimization
• Experience building modern UI components with React or Angular
• Strong computer science fundamentals, including distributed systems, algorithms, and concurrency

Responsibilities

• Own architecture and execution — Lead the technical direction of a real-time, multi-service clinical platform on GCP, while personally building the most critical and performance-sensitive components.
• Build production-grade LLM systems — Design and operate scalable LLM pipelines including prompt engineering, evaluation, retrieval/vector search, OCR-based document understanding, grounding, guardrails, and hallucination mitigation.
• Deliver real-time, high-performance systems — Architect distributed, event-driven, streaming services that support sub-second clinician experiences under heavy load, with strict SLOs and strong fault tolerance.
• Operationalize machine learning — Lead deployment and lifecycle management of models on Vertex AI, including feature pipelines, model registries, drift detection, A/B and shadow testing, and CI/CD-integrated quality controls.
• Design for scalability and change — Build flexible APIs, microservices, and data contracts that support rapid iteration, backed by strong testing, observability, and progressive delivery practices.
• Raise the engineering bar — Drive high-quality code reviews, establish standards across testing, CI/CD, security, and observability, and mentor engineers across full-stack and ML disciplines.
• Own outcomes end-to-end — Translate ambiguous clinical and business needs into delivered solutions, monitor real-world performance, and continuously iterate based on data.
• Partner across disciplines — Collaborate with Data Science, Product, Clinical SMEs, AI Governance, and Security teams to ensure solutions meet enterprise standards for compliance, privacy, and safety.

Skills

Python

Distributed systems

GCP (Vertex AI)

REST APIs

React/Angular

CI/CD

ML/Ops

LLM pipelines

Monitoring/Observability

Education

Bachelor's degree in CS or related

Master's or PhD in CS/ML a plus

Tools

Vertex AI

GKE

Cloud Run

Pub/Sub

BigQuery

IAM

Job description

CVS Health is actively hiring a Principal Software Engineer to lead the technical direction of a real-time clinical decision-support platform on Google Cloud. You will personally build critical, performance-sensitive components while mentoring engineers across UI, backend, and ML disciplines.

The role requires 10+ years in software engineering, hands-on staff/leadership, and extensive experience with Python, distributed systems, and GCP-based ML workflows.