Databricks Data Engineer

LanceSoft, Inc.; Toronto; On-site; CAD 170,000 - 210,000; Full time

Job summary

LanceSoft, Inc. in Toronto seeks a Senior Data Engineer with proven leadership to design and build scalable enterprise data platforms supporting Advisor360, CRM, ETF, and Mutual Fund data solutions.

You will lead Databricks-based pipelines, implement data models, and collaborate with analytics and AI initiatives to drive innovation and value. The role requires architectural insight and autonomous execution.

Qualifications

• 12+ years of experience in data engineering or large-scale data systems.
• Expert knowledge of AWS data services and the Databricks ecosystem.
• Strong SQL, Python, ETL/ELT experience and production deployments.

Responsibilities

• Design, develop, and optimize scalable data pipelines using Databricks and AWS.
• Lead data ingestion, transformation, and modeling for enterprise platforms.
• Define robust data models supporting analytics, reporting, and AI/ML.
• Translate complex business requirements into scalable technical solutions.
• Establish data quality, monitoring, testing, and operational practices.
• Mentor engineers and lead end-to-end solution delivery.
• Support AI readiness, data unification, metadata management, and enterprise integration.

Skills

12+ years exp

AWS Data Services

Databricks / Spark

Data modeling

SQL

Python

CI/CD / DevOps

Education

Bachelor's degree in Computer Science or related field

Tools

Databricks

Spark

AWS (S3, Glue, Lambda, Redshift)

Job description

We are seeking a highly experienced Senior Data Engineer with strong expertise in Databricks, AWS, modern data architecture, and data modeling to design and build scalable enterprise data platforms. The role will lead the development of data pipelines, analytics assets, and data foundations supporting platforms such as Advisor360, CRM, ETF, and Mutual Fund data solutions. This is a senior-level, autonomous role requiring strong technical leadership, consultative problem-solving, and architecture expertise.

Key Responsibilities

• Design, develop, and optimize scalable data pipelines using Databricks (PySpark, Delta Lake, Unity Catalog, Lakeflow) and AWS (S3, Glue, Lambda, Step Functions, Redshift)
• Lead data ingestion, transformation, and modeling initiatives for enterprise data platforms.
• Define and implement robust data models supporting analytics, reporting, and AI/ML use cases.
• Gather and translate complex business requirements into scalable technical solutions.
• Establish data quality, monitoring, testing, and operational best practices across data platforms.
• Mentor engineers, drive architecture standards, and lead end-to-end solution delivery.
• Support strategic initiatives including AI readiness, data unification, metadata management, and enterprise integration programs.

Required Skills & Experience

• 12+ years of experience in Data Engineering, Data Architecture, or large-scale distributed data systems.
• Expert knowledge of AWS Data Services and Databricks/Spark ecosystem.
• Strong expertise in data modeling (Dimensional, Canonical, Data Vault, Domain-Driven).
• Advanced SQL and Python development skills with ETL/ELT experience.
• Experience with CI/CD, GitHub, DevOps practices, automated testing, and production deployments.
• Proven ability to work independently and lead solutions in ambiguous business environments.

Preferred Qualifications

• Experience in Asset Management, Wealth Management, or Financial Services.
• Knowledge of data quality frameworks, metadata management, dbt, semantic layers, or data mesh concepts.
• Bachelor's degree in Computer Science, Engineering, Information Systems, or related field (or equivalent experience).