Principal AI Architect

Oscar; United States; Remote; USD 270,000 - 330,000; Full time

Job summary

Oscar is seeking a Principal AI Architect to own the AI technical function, define the architecture, and drive the quality and cost of in‑house AI systems. This hands‑on IC role leads from design through production, evolving toward Head of AI Engineering as the team grows.

You will build a centralized lakehouse‑based AI stack, govern models and deployments, and guide the organization on ROI, security, and scalable AI delivery. Fully remote anywhere in the United States.

Qualifications

• 10+ years in software, data, or ML engineering.
• Strong Python and SQL; builds and debugs AI systems.
• Production experience with LLM agents, RAG, and knowledge graphs.
• Proven ROI and AI cost control; deterministic vs LLM design skills.
• Experience securing AI systems in regulated environments.

Responsibilities

• Set the AI technical vision and roadmap with executive leadership.
• Decide deterministic vs dynamic components in system design.
• Design multi-agent architectures, tool use, and safety gates.
• Build end‑to‑end RAG, embeddings, and grounded citations.
• Develop knowledge graphs and ontology‑driven retrieval.
• Oversee document intelligence across contracts and payments.
• Lead data lakehouse design and governance for AI workloads.
• Select models by task, latency, cost, and data constraints.
• Deploy and operate AI services on managed or self‑hosted infra.
• Establish evaluation, monitoring, and drift control tied to outcomes.

Skills

Python

SQL

LLM agents

RAG

Knowledge graphs

Cost/ROI optimization

Regulated data security

Job description

We are working with a leading healthcare revenue cycle technology company that is looking for a Principal AI Architect to join its team. This is a newly created, hands‑on individual contributor role that owns the AI technical function at the company: the technical vision, the architecture, and the quality and cost of everything the company builds with AI. The successful candidate will be the company's deepest AI technical authority, building its AI, automation, and intelligence capabilities in‑house on a centralized, modern lakehouse data platform. The role is expected to evolve into Head of AI Engineering as the team grows.

Key Responsibilities

• Set the technical vision and roadmap with the executive leading AI and automation, giving leadership a clear view of feasibility, value, cost, and risk.
• Decide what should be deterministic (rules, parsers, schema validation, SQL, classical ML) and what should be dynamic (LLM reasoning, agentic tool use), using LLMs where inputs are ambiguous and plain code everywhere else.
• Design single- and multi‑agent systems: routing, planner/executor patterns, tool calling and MCP, state and memory, durable execution, retries, idempotency, human approval gates, and tool permission scoping.
• Build and tune RAG end to end: parsing, chunking strategy, embedding selection and fine‑tuning, hybrid search, re‑ranking, query rewriting, vector search, and grounded citations.
• Build knowledge graphs and ontology‑driven retrieval, including entity resolution and GraphRAG‑style hybrids, in partnership with the data organization.
• Handle document intelligence for contracts, payment files, and correspondence: OCR, layout‑aware parsing, table and field extraction, and multimodal models.
• Design the data layout AI depends on: medallion architecture, Delta table design, Unity Catalog governance and lineage, and feature and serving tables.
• Select models by task, accuracy, latency, cost, and data‑handling constraints. Fine‑tune where it pays off (SFT, LoRA/QLoRA, preference tuning) and distill into smaller, cheaper models.
• Deploy and serve on managed model serving and self‑hosted inference such as vLLM, using quantization, batching, autoscaling, and routing. Run the lifecycle in MLflow with shadow, canary, and A/B rollouts.
• Build evaluation and monitoring tied to business outcomes: golden datasets, retrieval metrics, groundedness, LLM‑as‑judge calibrated to human labels, release‑gating regression suites, and production drift monitoring.
• Own token and compute spend: routing and cascading, prompt and semantic caching, context compression, batch inference, and cost attribution per agent and workflow.
• Defend against prompt injection, tool‑based data exfiltration, and over‑permissioned agents with least privilege, guardrails, PII/PHI redaction, and full audit logging.
• Lead the technical build from design through production, writing the specs, patterns, and standards everyone building AI at the company works to, and mentor the AI engineering team that forms around this role.

Required Skills & Experience

• 10+ years in software, data, or ML engineering, with 7+ years in machine learning and increasing scope.
• Strong current Python and SQL. Builds, tests, and debugs AI systems personally.
• Production experience across LLM agents, RAG, knowledge graphs, closed and open‑weight models, fine‑tuning, and deployment.
• A record of proving ROI and controlling AI cost, and of choosing deterministic versus LLM‑driven design well.
• Experience securing AI systems in regulated or sensitive‑data environments.
• A record of setting technical direction others executed, including across employees, contractors, and vendors.

Nice to Have

• Healthcare, healthcare IT, or revenue cycle management (RCM) background, including HIPAA and PHI‑compliant AI deployment.
• Knowledge graph or ontology experience; document intelligence and claims or contract extraction.
• Azure, event‑driven architecture, and workflow automation or RPA experience.
• Experience standing up an AI Center of Excellence or internal AI platform.
• Deep Databricks and MLflow experience, including Delta, Unity Catalog, Model Serving, and Vector Search.

Why Consider This Opportunity?

• Base salary of $300,000, plus bonus and equity.
• Fully remote role, anywhere in the US.
• Rare opportunity to be the first and deepest AI technical authority at a growing healthcare technology company, building its AI capabilities from the ground up.
• Genuine ownership of the technical vision, architecture, and quality of everything the company builds with AI.
• Clear path to Head of AI Engineering as the team grows around you.
• Greenfield building on a modern lakehouse platform, with the chance to shape how AI is done across the entire business.