Senior Lead Data & AI Engineer

Vanguard; Malvern (Chester County); On-site; USD 130,000 - 160,000; Full time

Job summary

Vanguard is seeking a Lead Data & AI Engineer with 9 to 12+ years of experience to architect and deploy AI solutions. In this role, you will focus on Databricks and AWS environments to manage data engineering and AI orchestration frameworks.

Key responsibilities include building metadata-driven frameworks, automating AI pipelines, and ensuring real-time data processing. The ideal candidate has deep knowledge of cloud ecosystems and a track record in AI deployment.

Qualifications

• 9–12+ years in data engineering, specializing in AI deployment within cloud ecosystems.
• Hands-on experience deploying AI agents using RAG and agent orchestration frameworks.
• Proficient in AWS AI/ML services and orchestration tools.
• Strong knowledge of lakehouse architecture and data modeling best practices.
• Deep experience in data orchestration and scalable AI-driven workflows.

Responsibilities

• Architect reusable data and AI engineering frameworks using Databricks.
• Deploy Delta Live Tables and automate AI pipelines on Databricks.
• Integrate real-time data stream processing with AI agents.
• Ensure seamless orchestration between AI models and data pipelines.
• Manage AI agent lifecycles using tools like SageMaker and AWS services.

Skills

Data engineering

AI deployment

Cloud ecosystems

AWS services

Databricks

Tools

SageMaker

AgentOps

Unity Catalog

AWS Glue

Job description

Join us in a high-impact, high-visibility role where you will pioneer world-class Data & AI engineering solutions, building the next generation of intelligent, agent-driven systems that power real-time business decisions. We are seeking an expert Lead Data & AI Engineer with 9 to 12+ years of experience to architect and deploy AI agents into business workflows, focusing on Databricks and AWS environments. You will lead data engineering, AI agent orchestration, and scalable, production-grade AI architectures.

Key Responsibilities

• Architect and build reusable, metadata-driven data and AI engineering frameworks that standardize ingestion, transformation, feature engineering, and AI workflow deployment. Leverage Databricks, lakehouse architecture, declarative pipelines, and cloud-native services to enable scalable, governed, and reusable data products across the organization.
• Architect and deploy Delta Live Tables and Lakeflow jobs on Databricks to automate data processing, AI pipelines, and agent data refresh cycles.
• Leverage Databricks Workflows and Job Orchestration to schedule and monitor AI agent deployments across multiple business workflows.
• Integrate Lakeflow for real-time data stream processing, ensuring AI agents are updated and responsive to live data.
• Ensure seamless orchestration between AI models and data pipelines, using event-driven architectures for real-time inference and deployment.
• Implement and orchestrate AI agents using frameworks such as Agentic systems, AgentOps tooling, and solutions like Agents on Databricks (Agent-bricks).
• Hands‑on experience deploying AI agents using RAG, Graph RAG, MCP-enabled integrations, and agent orchestration frameworks such as AgentOps, AgentBricks, LangGraph, or cloud-native orchestration services.
• Manage AI agent lifecycles, monitoring, and scaling using tools like SageMaker, Bedrock, or AI orchestration frameworks on AWS.
• Ensure robust data governance, metadata management, and AI observability through Unity Catalog, AWS Glue, or custom metadata layers.
• Design for scalability and modularity, ensuring AI agents are reusable across multiple business processes.

Qualifications

• 9–12+ years in data engineering, specializing in AI deployment within cloud ecosystems (Databricks, AWS).
• Hands‑on experience deploying AI agents using frameworks like RAG, graph RAG, and orchestrating agents (AgentOps, Agent-bricks, etc.).
• Proficient in AWS AI/ML services (SageMaker, Bedrock) and orchestration tools (MWAA, Step Functions).
• Strong knowledge of lakehouse architecture, Unity Catalog, and data modeling best practices.
• Deep experience in data orchestration, monitoring, and scalable AI-driven workflows.

Sponsorship

Vanguard is not offering visa sponsorship for this position.