Principal Software Engineer

CVS Health; Annapolis (MD); On-site; USD 144,200 - 288,400; Full time

Job summary

CVS Health is seeking a Principal Software Engineer to lead a real-time, multi‑service clinical platform on Google Cloud. You will personally build critical components, design scalable APIs, and mentor engineers across UI, backend, and ML.

You will deliver production‑grade LLM pipelines, implement robust CI/CD, observability, and strong fault tolerance while ensuring clinical impact and high reliability of systems.

Qualifications

• 10+ years software engineering delivering production systems at scale.
• 5+ years as hands-on Staff or Principal-level engineer.
• Experience with real-time distributed systems with latency requirements.
• Proficient in Python including performance tuning and concurrency.
• Experience with major public clouds (GCP preferred).
• Strong CS fundamentals and API design experience.
• 3+ years operating production ML systems and LLM-powered apps.

Responsibilities

• Own architecture and execution of a real-time multi-service platform on GCP.
• Build production‑grade LLM systems with prompt engineering and evaluation.
• Deliver high‑performance, fault-tolerant distributed services with sub-second latency.
• Operationalize ML with Vertex AI, feature pipelines, drift detection, and CI/CD checks.
• Design scalable APIs, microservices, and robust data contracts.
• Lead code reviews, establish standards for testing, security, and observability.

Skills

Python

Distributed systems

GCP Vertex AI

LLM systems

CI/CD

React/Angular

Latency requirements

Education

Bachelor's in CS or related

Master's/PhD in CS or ML (plus)

Tools

Vertex AI

GKE

Cloud Run

Pub/Sub

BigQuery

Job description

We're building a world of health around every individual - shaping a more connected, convenient and compassionate health experience. At CVS Health®, you'll be surrounded by passionate colleagues who care deeply, innovate with purpose, hold ourselves accountable and prioritize safety and quality in everything we do. Join us and be part of something bigger - helping to simplify health care one person, one family and one community at a time.

Position Summary

We are hiring a Principal Software Engineer - a deeply hands‑on technical leader who thrives on solving complex problems, writing production code, and elevating the engineers around them. You will serve as the senior engineering voice across a portfolio of mission‑critical clinical decision‑support applications. These systems combine low‑latency distributed architectures with advanced LLM, OCR, and ML pipelines running on Google Cloud. This is not an "architect‑from‑the‑sidelines" role. You will actively design systems, write critical code, ship to production, support live environments, and mentor engineers across UI, backend, and ML. Success in this role is measured by the quality, performance, reliability, and clinical impact of what you deliver—not by presentations.

What You’ll Do

• Own architecture and execution — Lead the technical direction of a real‑time, multi‑service clinical platform on GCP, while personally building the most critical and performance‑sensitive components.
• Build production‑grade LLM systems — Design and operate scalable LLM pipelines (e.g., Claude, Gemini, GPT‑4‑class) including prompt engineering, structured outputs, evaluation frameworks, retrieval/vector search, OCR‑based document understanding, grounding, guardrails, and hallucination mitigation.
• Deliver real‑time, high‑performance systems — Architect distributed, event‑driven, streaming services that support sub‑second clinician experiences under heavy load, with strict SLOs and strong fault tolerance.
• Operationalize machine learning — Lead deployment and lifecycle management of models on Vertex AI, including feature pipelines, model registries, drift detection, A/B and shadow testing, and CI/CD‑integrated quality controls.
• Design for scalability and change — Build flexible APIs, microservices, and data contracts that support rapid iteration, backed by strong testing, observability, and progressive delivery practices.
• Raise the engineering bar — Drive high‑quality code reviews, establish standards across testing, CI/CD, security, and observability, and mentor engineers across full‑stack and ML disciplines.
• Own outcomes end‑to‑end — Translate ambiguous clinical and business needs into delivered solutions, monitor real‑world performance, and continuously iterate based on data.
• Partner across disciplines — Collaborate with Data Science, Product, Clinical SMEs, AI Governance, and Security teams to ensure solutions meet enterprise standards for compliance, privacy, and safety.

Required Qualifications

• 10+ years of software engineering experience with a strong record of delivering production systems at scale
• 5+ years as a hands‑on Staff or Principal‑level engineer (actively coding, not removed from implementation)
• Expert proficiency in Python including performance tuning, concurrency, and system optimization
• 5+ years building real‑time distributed systems (event‑driven, streaming, Pub‑Sub) with clear latency and reliability requirements
• 5+ years working in a major public cloud (GCP preferred: Vertex AI, GKE, Cloud Run, Pub/Sub, BigQuery, IAM, etc.)
• Deep experience with microservices architecture, API design (REST), and resilient systems (retries, backpressure, tracing)
• 3+ years building and operating production ML systems (MLOps, monitoring, retraining, deployment pipelines)
• 3+ years developing LLM‑powered applications in production, including prompt design, RAG/vector search, evaluation frameworks, and performance optimization
• Experience building modern UI components with React or Angular
• Strong computer science fundamentals, including distributed systems, algorithms, and concurrency

Preferred Qualifications

• Experience operating LLM systems in regulated environments (PII/PHI, governance, security, auditability)
• Familiarity with vector databases (e.g., Vertex Vector Search, Pinecone, Weaviate, pgvector) and advanced retrieval techniques
• Experience designing evaluation frameworks for LLM systems (gold datasets, regression testing, LLM‑as‑judge)
• Strong background in CI/CD, SRE practices, and SLO‑driven engineering
• Experience with healthcare data standards (FHIR, HL7) and compliance frameworks (HIPAA, HITRUST)
• Strong communication skills, with the ability to explain complex systems to both technical and non‑technical audiences
• Proven track record of mentoring engineers into senior and staff roles

Education

• Bachelor's degree in Computer Science or a related field (or equivalent experience)
• Master's or PhD in Computer Science, Machine Learning, or a related discipline is a plus

Pay Range

The typical pay range for this role is: $144,200.00 - $288,400.00. This pay range represents the base hourly rate or base annual full‑time salary for all positions in the job grade within which this position falls. The actual base salary offer will depend on a variety of factors including experience, education, geography and other relevant factors. This position is eligible for a CVS Health bonus, commission or short‑term incentive program in addition to the base pay range listed above. This position also includes an award target in the company's equity award program.

Our people fuel our future. Our teams reflect the customers, patients, members and communities we serve and we are committed to fostering a workplace where every colleague feels valued and that they belong.

Benefits

Great benefits for great people. We take pride in offering a comprehensive and competitive mix of pay and benefits that reflects our commitment to our colleagues and their families. This full‑time position is eligible for a comprehensive benefits package designed to support the physical, emotional, and financial well‑being of colleagues and their families. The benefits for this position include medical, dental, and vision coverage, paid time off, retirement savings options, wellness programs, and other resources, based on eligibility. Additional details about available benefits are provided during the application process and on Benefits Moments.

We anticipate the application window for this opening will close on: 10/02/2026.

Qualified applicants with arrest or conviction records will be considered for employment in accordance with all federal, state and local laws.