Principal Engineer – AI/ML Architecture

Jobtailor; Foster City (CA); On-site; USD 180,000 - 260,000; Full time

Job summary

Jobtailor seeks a senior AI systems architect to lead the end-to-end design of model training, evaluation, and deployment across our enterprise AI platform.

You will evaluate open-weight, commercial, and hybrid approaches, develop evaluation frameworks, and drive collaborations with AI infra providers. The role includes mentoring engineers across the US and India and aligning with product and cloud operations teams.

Qualifications

• 15+ years of software engineering experience with 5+ years in ML/AI systems.
• Experience training or fine-tuning large language models and shipping in production.
• Deep knowledge of transformer architectures and training optimization (LoRA, QLoRA, PEFT, RLHF, DPO).
• Experience with distributed training on GPU clusters.
• Strong understanding of RAG architectures, vector search, embeddings, and knowledge graphs.
• Hands-on experience with cloud AI/ML services for model hosting or training.
• Experience designing and running custom evaluation suites for LLMs.
• Proficiency in Python, PyTorch, and ML infrastructure tooling.
• Advanced degree in CS or ML (or equivalent).
• Experience with enterprise B2B SaaS platforms preferred.

Responsibilities

• Define the architecture for model training, evaluation, and serving across Coupa's AI platform.
• Evaluate and select model approaches against enterprise accuracy and cost requirements.
• Design evaluation frameworks that measure model quality on Coupa's task categories.
• Drive technical partnership evaluations with AI infrastructure and model providers.
• Architect training data pipelines, including synthetic data generation and quality validation.
• Design retrieval-augmented generation (RAG) systems with structured knowledge retrieval.
• Establish technical standards for model safety, tenant data isolation, and responsible AI.
• Write code, review PRs, and prototype approaches, especially in early phases.
• Mentor ML and data engineers across US and India.
• Collaborate with product, AI platform, and cloud operations teams.

Skills

ML Systems Architecture

Large Language Model Training

Transformer Architectures

Cloud AI/ML Services

Python and PyTorch

Education

Advanced Degree in Computer Science

Advanced Degree in Machine Learning

Tools

GPU Clusters

AI Infrastructure Tooling

Job description

• Define the architecture for model training, evaluation, and serving across Coupa's AI platform.
• Evaluate and select model approaches (open-weight, commercial, and hybrid) against enterprise accuracy and cost requirements.
• Design evaluation frameworks that measure model quality on Coupa's specific task categories.
• Drive technical partnership evaluations with AI infrastructure and model providers.
• Architect training data pipelines, including synthetic data generation and quality validation.
• Design retrieval-augmented generation (RAG) systems that extend our existing RAG infrastructure with structured knowledge retrieval.
• Establish technical standards for model safety, tenant data isolation, and responsible AI.
• Write code, review PRs, and prototype approaches, especially in the early phases.
• Mentor and guide ML and data engineers across US and India.
• Collaborate with product, existing AI platform, and cloud operations teams.

Requirements

• 15+ years of software engineering experience, with 5+ years focused on ML/AI systems.
• Demonstrated experience training or fine-tuning large language models. Must have shipped a fine-tuned or domain-adapted model to production.
• Deep knowledge of transformer architectures, training optimization (LoRA, QLoRA, PEFT, RLHF, DPO), and inference serving.
• Experience with distributed training on GPU clusters.
• Strong understanding of RAG architectures, vector search, embedding models, and knowledge graph integration.
• Hands‑on experience with cloud AI/ML services (model hosting, managed training, or equivalent).
• Experience designing and running custom evaluation suites for LLMs.
• Proficiency in Python, PyTorch, and ML infrastructure tooling.
• Advanced degree in Computer Science, Machine Learning, or equivalent practical experience.
• Experience with enterprise B2B SaaS platforms preferred.

Core Competencies

Expertise in architecting and optimizing machine learning systems, particularly in training and deploying large language models. Strong capability in designing evaluation frameworks and ensuring model safety and compliance within AI platforms.

Highest-signal resume keywords

• Machine Learning Systems Architecture
• Large Language Model Training
• Transformer Architectures
• Cloud AI/ML Services
• Python and PyTorch Proficiency

ATS Optimization Keywords

Hard Skills

• Model Training
• Model Evaluation
• Distributed Training
• Training Optimization
• RAG Systems Design
• Synthetic Data Generation
• Knowledge Graph Integration
• Embedding Models
• Evaluation Suite Design
• Fine‑Tuning Models

Soft Skills

• Mentoring
• Collaboration

Certifications & Qualifications

• Advanced Degree in Computer Science
• Advanced Degree in Machine Learning

Industry Keywords

• B2B SaaS Platforms
• Responsible AI
• Technical Standards
• Model Safety
• Tenant Data Isolation

Tools & Technologies

• GPU Clusters
• AI Infrastructure Tooling