Lead Data Engineer

Insight Global; North Vancouver; On-site; CAD 100,000 - 150,000; Full time

Job summary

Insight Global is seeking a Sr Data Engineer for their retail client in Vancouver. The role involves architecting data platforms, mentoring teams, and ensuring data quality across operations. Candidates should have significant experience in data engineering and leadership, especially with Azure and Databricks.

Qualifications

• 6-10 years of experience in data engineering, with 3+ years in leadership roles.
• Proven track record in designing large-scale data architectures.
• Strong Azure experience with Databricks and pipeline orchestration tools.

Responsibilities

• Architecting and implementing scalable data platforms using Azure, Databricks.
• Overseeing Unity Catalog adoption for centralized data governance.
• Leading and mentoring a team of data engineers.

Skills

Data Engineering

Azure

Databricks

SQL

Python

Pyspark

Infrastructure-as-code

Leadership

Tools

Terraform

ARM templates

Job description

Job Description

Insight Global is seeking a Sr Data Engineer to join a large retail client based in Vancouver BC. The data engineering team builds and maintains the data infrastructure that powers analytics, reporting, and data-driven decision making across the companys digital platforms, retail operations, and business units. We work with cross-functional agile squads to develop reliable data pipelines, improve data quality, build self-serve analytics tools for our business stakeholders, and ensure timely access to accurate data for our analysts and data scientists.

Day to Day Activities:

Architecting and implementing scalable data platforms using Azure, Databricks, and NoSQL databases

Overseeing Unity Catalog adoption for centralized data governance and metadata management

Designing integrations with ML infrastructure and collaborating with data science teams on model deployment

Leading and mentoring a team of data engineers, balancing hands-on coding with strategic oversight

Establishing coding standards, DevOps/SDLC workflows, and performance optimization practices

Driving backlog prioritization, technical planning, and cross-functional alignment

Implementing enterprise-grade data governance frameworks, including security (SecOps) and cost management (FinOps)

Championing data quality initiatives and advanced monitoring solutions

Evaluating emerging technologies (e.g., real-time processing, GenAI pipelines) for business impact

Partnering with business leaders to translate requirements into technical roadmaps

Leading hiring processes, performance evaluations, and team development

Exemplifying the companys leadership code (Lead Self, Lead Others, Lead Business) through innovation, accountability, and mentorship

Here are some of the things you could be working on in the future:

Leading the technical implementation of data mesh architecture and domain-oriented data products

Designing and implementing real-time data streaming solutions for critical business processes

Establishing advanced MLOps practices to operationalize machine learning workflows at scale

Developing comprehensive data observability solutions across the entire data ecosystem

Creating specialized data acceleration frameworks to improve platform performance

We are a company committed to creating inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity employer that believes everyone matters. Qualified candidates will receive consideration for employment opportunities without regard to race, religion, sex, age, marital status, national origin, sexual orientation, citizenship status, disability, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to Human Resources Request Form (https://airtable.com/app21VjYyxLDIX0ez/shrOg4IQS1J6dRiMo) . The EEOC "Know Your Rights" Poster is available here (https://www.eeoc.gov/sites/default/files/2023-06/22-088_EEOC_KnowYourRights6.12ScreenRdr.pdf) .

To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/ .

Skills and Requirements

Must Haves:

6-10 years of experience in data engineering, with 3+ years in leadership roles

Strong Azure experience with Databricks, and pipeline orchestration tools (Spark)

Strong experience with Unity Catalog, DLT and Asset bundles

Advanced skills in SQL, Python, Pyspark and infrastructure-as-code (Terraform, ARM templates)

Proven track record in designing large-scale data architectures (batch/streaming, lakehouse, NoSQL)

Strong leadership skills with experience mentoring teams, supporting Jr Engineers and driving technical excellence

Can communicate complex technical strategies to executives and stakeholders Plusses:

AWS Experience (they dont use AWS just yet but potentially next year)

Retail experience in the past null

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal employment opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment without regard to race, color, ethnicity, religion,sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military oruniformed service member status, or any other status or characteristic protected by applicable laws, regulations, andordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or the recruiting process, please send a request to [e-mail].