Principal Engineer

your Jared

Northern, San Diego (KY, CA)

Hybrid

USD 180,000 - 240,000

Full time

4 days ago

Be an early applicant

Application generator

A complete application in a minute — tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Fully remote, work from home

Employee Share Option Plan

Flexible working hours

Paid Time-Off

Periodic in-person offsites globally

Continued education support

Advancement opportunity

Job summary

ProveAI in San Diego seeks a senior AI/ML Engineer to lead end-to-end strategy, architecture, and productionization of ML/LLM workloads. You will own model deployment, observability, and scalable data pipelines across distributed systems, guiding cross-functional teams to reliable, safe, and cost-effective AI capabilities.

The role demands 10+ years of software/AI experience, deep ML expertise, and strong leadership to drive enterprise-grade GenAI initiatives.

Qualifications

10+ years of software engineering with recent hands-on AI/ML development.
Bachelor’s degree in CS or related field.
Deep technical expertise in ML, LLMs, transformers, and modern AI frameworks.
Production AI/ML deployment experience at scale.
Strong Python; Java/C++/JS a plus.
Experience with data engineering workflows and scalable pipelines.
Expertise with AWS/GCP/Azure, containers, Kubernetes, and distributed systems.
Hands-on AI/MLOps: deployment, monitoring, CI/CD, experiment tracking.
Proven leadership managing 10+ engineers and cross-functional teams.
Translate business needs into clear technical requirements and outcomes.
LLM productionization including finetuning, RAG, safety, evaluation.
Experience with Kubeflow, Vertex AI, SageMaker or similar platforms.
Background in model governance, drift detection, fairness evaluation, compliance.
Domain specialization in NLP, CV, recommender systems, or agentic systems.

Responsibilities

Define and own architecture for scalable AI/ML systems, incl. training, fine-tuning, inference, evaluation, monitoring.
Translate business needs into robust AI/ML designs and staged delivery plans.
Make strategic model choices and integration decisions for MLOps and safety.
Lead design reviews and technical decision-making across teams.
Build and deploy production-grade AI/ML/LLM models from concept to rollout.
Set standards for readiness, gates, drift, observability, and performance.
Partner with engineering to integrate models into distributed systems with SLOs.
Design data pipelines, feature stores, and data quality/traceability workflows.
Develop scalable AI/MLOps practices for automation of training and testing.
Evaluate AI/ML orchestration platforms and CI/CD for AI/ML workflows.
Own evaluation pipelines: latency, accuracy, cost, and model insights.
Instrument tracing and observability using telemetry standards.
Implement guardrails/safety systems for consistent behavior of AI features.
Collaborate with product/leadership to shape platform strategy and roadmap.
Provide trade-off analyses on performance, security, and maintainability.
Write technical docs and recommendations for decision-makers.
Mentor engineers in AI/ML best practices and governance.
Support hiring, leveling, and growth of the engineering team.

Skills

AI/ML development

Python

AWS/GCP/Azure

LLMs

Kubernetes

Distributed systems

Data pipelines

Model evaluation

Leadership

Transformers

Education

Bachelor’s degree in CS or related field

Master’s or PhD in CS/ML

Tools

Kubeflow

Vertex AI

SageMaker

MLflow

CI/CD for ML

Job description

San Diego

Job description

Drive the end-to-end technical strategy, architecture, and productionization of ProveAI’s machine learning systems, large language model (LLM) capabilities, and AI infrastructure. Own how models, evaluation pipelines, data workflows, and observability components are designed, deployed, monitored, and continuously improved to meet reliability, quality, safety, and cost goals. Provide deep AI/ML expertise and leadership across engineering teams, guiding model integration, AI/ML platform decisions, and scalable distributed systems that support enterprise-grade GenAI workloads.

Job requirements

10+ years of software engineering experience with significant recent hands‑on AI/ML/AI development.
Bachelor’s degree in CS or related field.
Deep technical expertise in machine learning, LLMs, transformers, and modern AI frameworks (PyTorch, TensorFlow, JAX, Scikit-learn).
Proven experience deploying production AI/ML or LLM systems at scale (not prototypes).
Strong programming expertise in Python; additional experience in Java, C++, or JavaScript is a plus.
Experience with data engineering workflows, feature stores, and scalable data pipelines.
Expertise with cloud platforms (AWS/GCP/Azure), containerization, orchestration (Kubernetes), and distributed systems.
Hands‑on AI/MLOps: model deployment, monitoring, CI/CD for AI/ML, experiment tracking, and evaluation frameworks.
Demonstrated technical leadership managing teams of 10+ engineers and influencing cross‑functional architecture.
Strong ability to translate ambiguous business needs into clear technical requirements and production outcomes.
Expertise with LLM productionization including finetuning, retrieval‑augmented generation (RAG), safety/guardrails, and evaluation.
Experience with AI/ML flow, Kubeflow, Vertex AI, SageMaker, or similar platforms.
Background in model governance, drift detection, fairness/bias evaluation, and compliance.
Domain specialization (NLP, computer vision, recommender systems, or agentic systems).

Nice to Have:

Master’s or PhD in Computer Science, Machine Learning, or related discipline.
Cloud platform expertise (AWS, GCP, Azure) with experience deploying AI/ML workloads at scale
Strong product mindset with ability to translate business requirements into technical solutions
Contributions to AIops/MLOps platforms (MLflow, Kubeflow, Vertex AI) and CI/CD for ML workflows
Domain expertise in specific AI application areas such as computer vision, NLP, or recommendation systems
Experience with model monitoring, drift detection, and model governance in production environments
Previous experience with AI observability and troubleshooting

Job responsibilities

Define and own the architecture for scalable AI/ML systems, including training, fine‑tuning, inference, evaluation, and monitoring pipelines.
Translate ambiguous business and product requirements into robust AI/ML system designs and staged delivery plans.
Make strategic decisions on model selection, LLM integrations, evaluation frameworks, model gateways, guardrails, and safety mechanisms.
Lead design reviews, architecture forums, and technical decision‑making across teams.
Build and deploy production‑grade AI/ML/LLM models, transformers, and generative AI features—from initial concept through production rollout.
Establish standards for model readiness, evaluation gates, rollout/rollback, drift detection, observability, and ongoing performance management.
Partner with engineering teams to integrate models into distributed systems with clear SLOs, telemetry, and error‑budget mechanisms.
Design and improve data pipelines, feature stores, and data quality/lineage workflows supporting model training and inference.
Develop scalable AI/MLOps/AIOps practices for automation of training, testing, deployment, and monitoring.
Evaluate and implement AI/ML workflow orchestration platforms (e.g., AI/MLflow, Kubeflow, Vertex AI) and CI/CD for AI/ML.
Own evaluation pipelines—latency, accuracy, cost, hallucination metrics, prompt versioning, and model performance insights.
Instrument tracing and model observability using best‑practice frameworks and telemetry standards.
Implement guardrails and safety systems to ensure consistent, controlled behaviour of LLM‑powered features.
Partner closely with product, engineering, and leadership to shape platform strategy and AI feature roadmap.
Provide trade‑off analyses that incorporate model performance, security, compliance, scalability, and long‑term maintainability.
Write clear technical documents, proposals, and mechanism‑based recommendations to guide executive decision‑making.
Mentor senior/junior engineers in AI/ML best practices, distributed systems, experimentation, and model governance.
Support hiring, leveling, performance feedback, and the growth of a high‑calibre engineering team.