AI Engineer

TechDigital Group; San Jose (CA); On-site; USD 140,000 - 210,000; Full time

Job summary

TechDigital Group is seeking an AI Engineer to lead ML sustenance across a multi-service portfolio, acting as the onsite technical point of contact and mentoring a globally distributed team.

You will design and sustain automated training pipelines, data ingestion, and feature engineering, implement robust evaluation, deployment, and observability, and drive cost-aware, compliant AI solutions.

Qualifications

• 8+ years in software, ML engineering, or MLOps/DevOps, incl. 2+ years in a tech lead role.
• Experience with GCP MLOps services (Vertex AI, Kubeflow) or open-source ecosystems (Airflow, MLflow).
• Proficiency in Python or Go and large-scale data processing.
• Experience with production deployments, container orchestration, and CI/CD pipelines.
• Experience with distributed systems monitoring, SLOs, on-call incidents, and root-cause analysis.
• Ability to lead and mentor engineers across time zones and communicate with stakeholders.

Responsibilities

• Set technical direction and roadmap for ML sustenance across a multi-service portfolio; mentor a globally distributed team.
• Design, build, and sustain automated continuous training, data ingestion, and feature engineering pipelines.
• Configure evaluation pipelines, golden datasets, and dashboards; set up alerting for drift or failures.
• Build and maintain CI/CD workflows for safe deployment and rollback of containerized ML models; ensure observability.
• Enforce data isolation, IAM/RBAC, and strict data retention/deletion policies across tenants.
• Develop AI-assisted automation to scale operations and optimize compute/token costs.

Skills

ML lifecycle

GCP Vertex AI

Python

Go

Kubernetes

Docker

CI/CD

Distributed systems

Stakeholder leadership

Tools

Kubeflow

MLflow

Apache Airflow

Jenkins

GitLab CI

Kubernetes

Docker

Job description

AI Engineer

Key Responsibilities:

• Technical Leadership & Stakeholder Management: Set the technical direction and roadmap for ML sustenance across a multi-service portfolio. Act as the primary onsite technical point of contact, leading and mentoring a globally distributed (follow-the-sun) team of engineers. Drive design reviews, agree on service-level objectives (SLOs), and partner with downstream client teams to onboard new AI agents and standardize model evaluation processes.
• ML/GenAI Architecture & Pipelines: Design, build, and sustain automated continuous training (CT), data ingestion, and feature engineering pipelines (utilizing concepts similar to Kubeflow, MLflow, or Apache Airflow). Ingest datasets from distributed SQL or messaging systems (like Apache Kafka). Establish robust versioning and governance for models, prompts, and evaluation configurations, setting up experiment tracking and hyperparameter tuning to optimize performance.
• Agentic Evaluation Infrastructure & Quality: Configure isolated/hermetic agent environments (using Kubernetes/Docker principles), integrating tools for synthetic data generation and simulated execution traces. Build and sustain evaluation pipelines, golden datasets, and quality dashboards to continuously monitor model output. Set up configurable alerting based on specific metric thresholds (e.g., Accuracy, AUC, Trajectory Exact Match, ROUGE) to proactively identify semantic drift or hallucinations, and audit automated evaluators (LLM-as-a-judge).
• Deployment, Serving, Reliability & Observability: Build and maintain CI/CD workflows for safe, automated deployment and rollback of containerized ML models. Set up comprehensive observability (metrics, logs, traces) to monitor production health and low-latency serving. Participate in operational support to triage incidents, mitigate production issues, manage traffic rollouts, and drive root-cause analysis to resolve capacity bottlenecks and lower MTTR.
• Privacy & Security Implementation: Enforce strict Data Isolation Models for sensitive data. Implement strict Identity and Access Management (IAM), Role-Based Access Control (RBAC), and adhere to rigorous data retention and deletion policies (aligned with Zero Trust and strict compliance standards) across all tenant onboardings and platform versions.
• Operational Automation & Cost Optimization: Develop AI-assisted operational automation (e.g., monitoring agents, auto-triage, and documentation-sync hooks) to scale the team's impact. Optimize compute and token costs across serving and evaluation workloads to ensure cost-efficient scaling of AI capabilities.

Minimum Qualifications:

• 8+ years of experience in software engineering, ML engineering, or MLOps/DevOps, focusing on the ML lifecycle (continuous training, model deployment, and continuous monitoring), including 2+ years in a technical lead role.
• Strong experience with Google Cloud Platform (GCP) MLOps services (Vertex AI, Kubeflow) or equivalent open-source ecosystems (Apache Airflow, MLflow).
• Proficiency in programming languages such as Python or Go, and experience with large-scale data processing and distributed systems.
• Experience working with large-scale production deployment, container orchestration (e.g., Kubernetes, Docker), and CI/CD pipelines (e.g., Jenkins, GitLab CI).
• Experience with production operations for distributed systems: monitoring and alerting, SLO definition, on‑call incident triage, and root‑cause analysis.
• Proven ability to design and implement end‑to‑end ML pipelines for data management, model training, evaluation, and deployment.
• Demonstrated ability to lead and mentor engineers across time zones, communicate clearly with technical and non‑technical stakeholders, and manage multiple partner priorities.

Preferred Qualifications:

• Familiarity with Large Language Models (LLMs) and open‑source Agentic AI frameworks (e.g., LangChain, AutoGen, LlamaIndex, or internal equivalent like Agent Development Kit).
• Knowledge of standard ML evaluation metrics for Generative and Classification models (e.g., BLEURT, ROUGE, Precision/Recall, AUC, Trajectory Exact Match).
• Experience managing highly sensitive data (e.g., PII/PHI) and implementing strict access control and encryption mechanisms.
• Strong analytical and debugging skills, with the ability to troubleshoot complex, distributed ML systems and infrastructure dependencies.
• Experience monitoring model quality in production, including data/concept drift detection, A/B testing, and building quality dashboards (e.g., Grafana, Datadog)
• Experience running knowledge-transfer and service‑transition programs, or working in a sustenance, support‑engineering, or SRE organizationa