AI Engineer Lead

Shrive Technologies; San Jose (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Shrive Technologies is seeking an experienced ML Tech Lead to steer the ML sustenance roadmap across a multi-service portfolio in San Jose, CA. You will mentor a follow-the-sun team, drive design reviews, and partner with downstream teams to onboard AI agents and standardize model evaluation.

The role requires building automated training pipelines, data ingestion, and governance with robust observability, alerting, and security practices.

Qualifications

• 8+ years of experience in software or ML engineering with a focus on ML lifecycle.
• 2+ years in a technical lead role and ability to mentor across time zones.
• Experience with large-scale production deployment and distributed systems.

Responsibilities

• Set the technical direction and roadmap for ML sustenance across a multi-service portfolio.
• Lead and mentor a globally distributed team of engineers and act as onsite tech lead.
• Drive design reviews, define SLOs, and onboard new AI agents with partner teams.
• Design, build and sustain automated CT, data ingestion and feature engineering pipelines.
• Establish model/version governance, experiment tracking and KPI monitoring.

Job description

Key Responsibilities:

Technical Leadership & Stakeholder Management: Set the technical direction and roadmap for ML sustenance across a multi-service portfolio. Act as the primary onsite technical point of contact, leading and mentoring a globally distributed (follow-the-sun) team of engineers. Drive design reviews, agree on service-level objectives (SLOs), and partner with downstream client teams to onboard new AI agents and standardize model evaluation processes. ML/GenAI Architecture & Pipelines: Design, build, and sustain automated continuous training (CT), data ingestion, and feature engineering pipelines (utilizing concepts similar to Kubeflow, MLflow, or Apache Airflow). Ingest datasets from distributed SQL or messaging systems (like Apache Kafka). Establish robust versioning and governance for models, prompts, and evaluation configurations, setting up experiment tracking and hyperparameter tuning to optimize performance. Agentic Evaluation Infrastructure & Quality: Configure isolated/hermetic agent environments (using Kubernetes/Docker principles), integrating tools for synthetic data generation and simulated execution traces. Build and sustain evaluation pipelines, golden datasets, and quality dashboards to continuously monitor model output. Set up configurable alerting based on specific metric thresholds (e.g., Accuracy, AUC, Trajectory Exact Match, ROUGE) to proactively identify semantic drift or hallucinations, and audit automated evaluators (LLM-as-a-judge). Deployment, Serving, Reliability & Observability: Build and maintain CI/CD workflows for safe, automated deployment and rollback of containerized ML models. Set up comprehensive observability (metrics, logs, traces) to monitor production health and low-latency serving. Participate in operational support to triage incidents, mitigate production issues, manage traffic rollouts, and drive root-cause analysis to resolve capacity bottlenecks and lower MTTR. Privacy & Security Implementation: Enforce strict Data Isolation Models for sensitive data. Implement strict Identity and Access Management (IAM), Role-Based Access Control (RBAC), and adhere to rigorous data retention and deletion policies (aligned with Zero Trust and strict compliance standards) across all tenant onboardings and platform versions. Operational Automation & Cost Optimization: Develop AI-assisted operational automation (e.g., monitoring agents, auto-triage, and documentation-sync hooks) to scale the team's impact. Optimize compute and token costs across serving and evaluation workloads to ensure cost-efficient scaling of AI capabilities.

Minimum Qualifications:

8+ years of experience in software engineering, ML engineering, or MLOps/DevOps, focusing on the ML lifecycle (continuous training, model deployment, and continuous monitoring), including 2+ years in a technical lead role. Strong experience with Google Cloud Platform (GCP) MLOps services (Vertex AI, Kubeflow) or equivalent open-source ecosystems (Apache Airflow, MLflow). Proficiency in programming languages such as Python or Go, and experience with large-scale data processing and distributed systems. Experience working with large-scale production deployment, container orchestration (e.g., Kubernetes, Docker), and CI/CD pipelines (e.g., Jenkins, GitLab CI). Experience with production operations for distributed systems: monitoring and alerting, SLO definition, on-call incident triage, and root-cause analysis. Proven ability to design and implement end-to-end ML pipelines for data management, model training, evaluation, and deployment. Demonstrated ability to lead and mentor engineers across time zones, communicate clearly with technical and non-technical stakeholders, and manage multiple partner priorities.

Preferred Qualifications:

Familiarity with Large Language Models (LLMs) and open-source Agentic AI frameworks (e.g., LangChain, AutoGen, LlamaIndex, or internal equivalent like Agent Development Kit). Knowledge of standard ML evaluation metrics for Generative and Classification models (e.g., BLEURT, ROUGE, Precision/Recall, AUC, Trajectory Exact Match). Experience managing highly sensitive data (e.g., PII/PHI) and implementing strict access control and encryption mechanisms. Strong analytical and debugging skills, with the ability to troubleshoot complex, distributed ML systems and infrastructure dependencies. Experience monitoring model quality in production, including data/concept drift detection, A/B testing, and building quality dashboards (e.g., Grafana, Datadog) Experience running knowledge-transfer and service-transition programs, or working in a sustenance, support-engineering, or SRE organization