Hybrid Lead Data & AI Architect (Databricks Lakehouse)

Vanguard; Toronto; Hybrid; CAD 165,000 - 200,000; Full time

Benefits offered by this job

Health insurance

Accident and life insurance

Retirement savings plans

Hybrid work model

Job summary

Vanguard Toronto is seeking a Senior Lead Data & AI Engineer to drive enterprise-scale data platforms, Databricks Lakehouse architecture, and AI-ready data products. You will lead design, coding, and governance while mentoring engineers and aligning analytics with business outcomes.

You will partner with data scientists and product teams to operationalize data products, implement CI/CD, and advance cloud-native modernization across the organization.

Qualifications

• 9-12+ years of experience in Data Engineering, Data Platform Engineering, or Analytics Engineering.
• 5+ years of hands-on experience building enterprise solutions on Databricks.
• Strong expertise in Lakehouse architecture, Delta Lake, Unity Catalog, and modern data engineering patterns.
• Proven experience building large-scale data pipelines using Spark, PySpark, SQL, and Python.
• Deep experience with AWS services supporting Data & AI platforms.
• Experience implementing Delta Live Tables, Lakeflow, Databricks Workflows, and metadata-driven engineering approaches.
• Strong understanding of data modeling, data products, data governance, metadata management, and data quality frameworks.
• Experience with CI/CD, Infrastructure-as-Code, automated testing, and cloud-native engineering practices.
• Demonstrated leadership experience mentoring engineers and leading complex technical initiatives.
• Strong communication and stakeholder management skills.

Responsibilities

• Lead the architecture, design, and delivery of enterprise-scale data engineering solutions leveraging Databricks and AWS.
• Design and implement scalable, resilient, and reusable data pipelines supporting batch, streaming, and event-driven workloads.
• Drive engineering best practices across data ingestion, transformation, orchestration, testing, monitoring, and deployment.
• Establish reusable frameworks, patterns, and standards to accelerate delivery and improve engineering quality across teams.
• Architect and implement modern Lakehouse solutions utilizing Databricks, Delta Lake, Unity Catalog, and cloud-native services.
• Develop metadata-driven and configuration-based frameworks that standardize ingestion, transformation, quality validation, and deployment processes.
• Design and implement Delta Live Tables (DLT), Lakeflow Pipelines, and Databricks Workflows to automate enterprise data processing.
• Build trusted, governed, and reusable data products that support analytics, reporting, data science, and AI initiatives.
• Optimize platform performance, scalability, reliability, and cost efficiency across large-scale workloads.
• Lead migration and modernization initiatives from legacy platforms to cloud-native data architectures.
• Implement data observability, monitoring, lineage, metadata management, and quality controls throughout the data lifecycle.
• Partner with architecture and platform teams to advance enterprise data platform capabilities and engineering standards.
• Support implementation of CI/CD, Infrastructure-as-Code, automated testing, and DevSecOps practices.
• Build AI-ready data foundations that support machine learning, Generative AI, and intelligent business applications.
• Partner with Data Scientists, AI Engineers, and business teams to operationalize data products and AI solutions.
• Support implementation of vector search, semantic models, retrieval-augmented generation (RAG), and knowledge-driven AI capabilities.
• Contribute to AI orchestration and agent-based solutions where business value can be accelerated through intelligent automation.
• Enable reliable integration between AI applications and enterprise data platforms.
• Implement enterprise-grade governance using Unity Catalog, metadata repositories, and data quality frameworks.
• Ensure compliance with enterprise security, privacy, and regulatory requirements.
• Design automated controls for data validation, observability, lineage, and operational monitoring.
• Partner with governance organizations to improve trust and accessibility of critical data assets.
• Act as a trusted technical advisor to business, analytics, and technology stakeholders.
• Mentor and provide technical leadership to Data Engineers and platform teams.
• Lead architecture reviews, design workshops, and engineering governance activities.
• Translate complex technical concepts into actionable business outcomes for leadership audiences.

Skills

Data Engineering

Databricks

Lakehouse

AWS

Architecture

Leadership

Mentoring

Python

PySpark

CI/CD

Education

Bachelor's or Master's in CS/Engineering

Tools

Delta Lake

Unity Catalog

Delta Live Tables

Lakeflow

Databricks Workflows

AWS Services

Job description

Vanguard Toronto is seeking a Senior Lead Data & AI Engineer to drive enterprise-scale data platforms, Databricks Lakehouse architecture, and AI-ready data products. You will lead design, coding, and governance while mentoring engineers and aligning analytics with business outcomes.

You will partner with data scientists and product teams to operationalize data products, implement CI/CD, and advance cloud-native modernization across the organization.