Sr. Data Engineer

Recruiting In Motion - Eastern Ontario; Toronto; Hybrid; CAD 120,000 - 140,000; Full time

Job summary

Recruiting In Motion - Eastern Ontario is seeking a Senior Data Engineer to design and scale our Azure Databricks Lakehouse platform in the Greater Toronto Area. This senior role focuses on end-to-end data lifecycle design, production pipelines, and governance across Bronze, Silver, and Gold layers.

You will enable AI pipelines, embeddings, and MCP/LLM integrations while collaborating with IT leadership and business stakeholders to drive enterprise analytics initiatives.

Qualifications

• Proven hands-on production experience with Azure Databricks Lakehouse architecture.
• Strong Python, PySpark, Spark SQL, and Delta Lake fundamentals.
• Experience building scalable ETL/ELT pipelines and data models.
• Excellent communication with technical and executive stakeholders.

Responsibilities

• Design, build, and maintain production data pipelines on the Azure Databricks Lakehouse.
• Implement Bronze/Silver/Gold data layers and oversee data governance with Unity Catalog.
• Optimize Databricks clusters and integrate Azure SQL Server for analytics workflows.
• Collaborate with business partners and AI teams on RAG, embeddings, and LLM pipelines.

Skills

Python

PySpark

Spark SQL

Delta Lake

Azure Databricks

SQL Server

Data pipelines

Data governance

Unity Catalog

Tools

Azure Databricks

Unity Catalog

Azure SQL Server

Job description

Job Title: Senior Data Engineer Location: Greater Toronto Area, ON (Hybrid: 4 days onsite / 1 day remote) Salary Range: $120,000 – $140,000 base salary + quarterly discretionary bonus

Position Overview

We are seeking a Senior Data Engineer to design, build, and maintain the production data and AI foundation on the Azure Databricks Lakehouse platform. Reporting to IT leadership, you will play a hands‑on role in scaling our enterprise data architecture (Medallion architecture across Bronze, Silver, and Gold layers), optimizing Databricks clusters hosted in Azure, and enabling next‑generation AI pipelines (RAG, vector search, LLM integrations, and custom MCP servers). This position requires a senior mindset—someone who brings deep hands‑on expertise with Azure Databricks and SQL Server, takes ownership of end‑to‑end data lifecycle design, and can communicate effectively with technical leads, business stakeholders, and executive leadership.

Key Responsibilities

Lakehouse Engineering & Databricks Operations

• Data Pipeline Development: Design, build, and optimize scalable ETL/ELT pipelines using Python, PySpark, Spark SQL, and Delta Lake.
• Medallion Architecture: Implement and maintain robust Bronze, Silver, and Gold data layers to ensure clean, structured, and audit‑ready data.
• Azure Databricks Infrastructure: Configure and manage Databricks clusters, notebook workflows, and Unity Catalog governance for access control and data lineage.
• Azure SQL Server Integration: Build, tune, and optimize database schemas, queries, data normalization models, and integration pathways with Azure SQL Server.

AI Enablement & Platform Architecture

• RAG & Vector Search: Build and support Retrieval‑Augmented Generation (RAG) pipelines over enterprise data, including document ingestion, embeddings generation, and vector database indexing.
• MCP & LLM Workflows: Build and maintain Model Context Protocol (MCP) servers and LLM API integrations to securely expose Lakehouse data to AI agents and business applications.
• Data Quality & Governance: Establish automated data quality checks, schema evolution standards, and performance optimization across all pipelines.

Stakeholder Engagement & Team Collaboration

• Act as a technical subject matter expert, bridging data engineering with business analytics and AI use cases.
• Translate complex data architectures into business context when interfacing with cross‑functional business units and stakeholders.

Qualifications & Competencies Mandatory Requirements

• Proven Hands‑On Experience: Direct, verifiable production experience building, deploying, and maintaining data solutions on Azure Databricks Lakehouse (theoretical knowledge or high‑level exposure alone will not suffice).
• Core Tech Stack: Advanced proficiency in Python, PySpark, SQL (Spark SQL), and Delta Lake.
• Governance: Hands‑on experience implementing data access controls, lineage, and governance using Unity Catalog.
• Database & Architecture: In‑depth understanding of data normalization, relational database modeling, query tuning, and integration patterns with Azure SQL Server.
• Communication: Exceptional verbal and written communication skills with the confidence and maturity to collaborate directly with business partners and executive stakeholders.

Work Requirement

Ability to work 4 days per week onsite at our office.

Strongly Preferred / AI Qualifications

• GenAI / RAG Infrastructure: Experience developing production RAG workflows (embeddings, vector search, retrieval evaluation).
• Model Context Protocol (MCP): Experience building or integrating custom MCP servers or connecting AI agents to enterprise data systems.
• LLM Integration: Hands‑on integration of large language model APIs (e.g., Azure OpenAI) into backend data pipelines.
• Microsoft Ecosystem: Familiarity with broader Azure data tools; Microsoft Fabric exposure is an asset.

AI may be used to support certain administrative aspects of our recruitment process. However, all applications and resumes are reviewed by qualified human reviewers, and all screening and hiring decisions are made solely by the hiring team. The posted salary range reflects expected compensation for this role in accordance with Ontario pay transparency requirements. Actual compensation will be determined based on the successful candidate’s experience, skills, and qualifications and may fall anywhere within the range.