Principal AI Researcher - Production-Grade LLM Systems

Innovaccer Inc.; San Francisco (CA); Hybrid; USD 190,000 - 240,000; Full time

Benefits offered by this job

Generous PTO: 20 days

Parental leave

Rewards & recognition

Medical, dental, vision benefits; life

Job summary

Innovaccer Inc. is seeking a Principal AI Researcher to build intelligent, production-grade AI systems at scale, collaborating with product, engineering, and data teams on LLM-based solutions, AI agents, retrieval-augmented generation, and automation workflows.

The candidate is hands-on across the full AI lifecycle, from prototyping to production deployment and optimization, and can explain results to non-technical stakeholders. This role offers strong growth and impact in healthcare AI.

Qualifications

• MS or PhD in Computer Science, Machine Learning, or a related quantitative field. Exceptional BS candidates with substantial research or open-source work will be considered.
• Depth beyond coursework: first-author publications at NeurIPS, ICML, ICLR, ACL, EMNLP, or similar; meaningful open-source ML contributions; a research internship at an AI lab; or models you trained and shipped that people actually used.
• You have fine-tuned an open-weight model yourself, understand the difference between parameter-efficient and full fine-tuning, and can explain why you chose one.
• Strong Python and PyTorch. Familiarity with the current training and serving stack (HuggingFace, FSDP or DeepSpeed, vLLM or SGLang, or equivalents).
• Some exposure to multi-GPU training, even at lab scale. You should know what a sharding strategy is and why it matters.
• Evidence you can finish things.
• You have post-trained models that ran in production and you know what broke.
• You have built the model layer behind a real product feature, including fine-tuning at more than one model size and composing multiple models into a system that met a production accuracy and latency bar.
• Hands-on multi-node training experience, tens of GPUs at minimum, on models in the tens of billions of parameters or larger.
• You have built or substantially owned a data pipeline that fed a real training run, including the unglamorous parts: dedup, filtering, decontamination, format normalization.
• You can scope an ambiguous problem into a plan and tell the difference between a research question and an engineering task.
• Mentoring or setting direction for other engineers is a plus and will matter more as the team grows.
• You have led post-training on models in the hundreds of billions of parameters, on runs spanning hundreds of GPUs, and you owned the outcome rather than a slice of it.
• You have designed the complete model architecture behind a shipped product, spanning small, medium, and large models, and made the tier and composition decisions that determined whether it worked economically.
• You have made the calls that decide whether a model program works: data mixture, objective design, when to stop a run, when to abandon an approach that a team is emotionally invested in.
• Deep familiarity with large-scale distributed training failure modes and the operational discipline that keeps a long run alive.
• A public or verifiable record: papers, model releases, systems, or a training program whose results speak for themselves.
• You can set a two-to-four-quarter technical direction and then get a team to execute it. You will effectively define what in-house modeling means at Innovaccer.

Responsibilities

• Take an idea from paper to prototype to production.
• Choose model sizes and compose several models into a working system, with a product-level accuracy bar.
• Write real code in Python and PyTorch, with systems awareness to improve training speed.
• Design experiments, state hypotheses, run ablations, and report objective results.
• Measure and validate model performance; avoid misleading results.
• Read current research and discern enduring techniques from fleeting trends.
• Explain work to non-AI stakeholders, including clinicians and operators.

Skills

Python

PyTorch

Distributed training

Model evaluation

Experiment design

Data pipelines

Production ML

Mentoring

Education

MS/PhD in Computer Science or ML

BS with research/open-source work

Tools

HuggingFace

DeepSpeed

FSDP

vLLM

SGLang

Job description

Innovaccer Inc. is seeking a Principal AI Researcher to build intelligent, production-grade AI systems at scale, collaborating with product, engineering, and data teams on LLM-based solutions, AI agents, retrieval-augmented generation, and automation workflows.

The candidate is hands-on across the full AI lifecycle, from prototyping to production deployment and optimization, and can explain results to non-technical stakeholders. This role offers strong growth and impact in healthcare AI.