Senior Consultant/Manager - Data Engineer

Electric Mind

Toronto

On-site

CAD 110,000 - 160,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Electric Mind Inc. is hiring a data engineer to build and maintain data pipelines on Databricks for a financial services data platform. You will transform data with PySpark, validate lineage, and publish curated data to downstream consumers via file exports and Kafka.

This role offers permanent and contract tracks, growth into a data engineering specialist, and collaboration with a senior engineer in a fast-paced, inclusive Toronto environment.

Qualifications

  • Solid Python fundamentals and ability to write clean, readable code.
  • Experience with Databricks or Delta Lake is a plus, plus SQL proficiency.

Responsibilities

  • Write and maintain PySpark transforms for silver and gold layer tables (customer, account, transaction data models).
  • Work with Lakeflow Declarative Pipelines and CDC/SCD patterns for data merges.
  • Query and validate data using SQL warehouses across environments.
  • Debug failed pipelines by tracing logs and fixing schema drift or quality issues.
  • Maintain reference/lookup tables and small utility scripts (PowerShell/Python).
  • Write and update unit/integration tests; participate in code reviews (GitLab).
  • Keep documentation current when changes are made.

Skills

Python
SQL
Git workflow
Debugging
Reading code
OOP basics

Tools

Databricks
Apache Spark
Azure Databricks
Delta Lake
GitLab
PowerShell

Job description

About the Role

We are hiring for both permanent and contract positions for this role. We're looking for a developer to help build and maintain data pipelines on Databricks for a financial services/wealth management data platform. The platform ingests data from multiple source systems, transforms it through a medallion architecture (bronze to silver to gold) using PySpark, and publishes curated data to downstream consumers via file exports and Kafka. You'll work alongside a senior engineer, writing transform code, fixing pipeline issues, and helping keep the environment healthy across dev/qa/uat.

This is a good fit for someone with solid Python fundamentals and some Spark/SQL exposure who wants to grow into a data engineering specialist. You won't be expected to know Databricks internals on day one - you will be expected to learn fast, ask good questions, and write clean code.

What You'll Work On
  • Writing and maintaining PySpark transforms (hand-written, not a generic framework) for silver and gold layer tables - things like customer, account, and transaction data models.
  • Working with Lakeflow Declarative Pipelines (Databricks' current pipeline framework - successor to DLT) and Auto CDC / SCD Type 2 patterns for change-data-capture merges.
  • Querying and validating data in Unity Catalog across environments (dev/qa/uat) using SQL warehouses.
  • Debugging failed pipeline runs: reading pipeline event logs, tracing bad records, fixing schema drift or data quality issues.
  • Maintaining reference/lookup tables and small utility scripts (PowerShell/Python) used to operate the platform.
  • Writing and updating unit/integration tests for transform logic.
  • Participating in code review, using Git feature branches and merge requests (GitLab).
  • Keeping documentation current when you change how something works.
Must-Have Skills
  • Python— comfortable writing clean, readable code; understands functions, modules, basic OOP.
  • Databricks
  • Some exposure to Apache Spark / PySpark, or strong SQL skills plus a willingness to learn Spark quickly.
  • Working knowledge of SQL (joins, aggregations, window functions).
  • Basic Git workflow: branches, commits, pull/merge requests.
  • Comfortable reading other people's code and stack traces, and debugging methodically (not guess-and-check).
Nice-to-Have Skills
  • Direct experience with Databricks (notebooks, jobs, clusters, or SQL warehouses).
  • Familiarity with Delta Lake, medallion architecture (bronze/silver/gold), or CDC/SCD concepts.
  • Exposure to Azure (this environment runs on Azure Databricks with Azure AD service principal auth).
  • Experience with Kafka or other streaming/event systems.
  • Experience with CI/CD pipelines (this project uses GitLab CI).
  • Prior work in financial services or a regulated data environment.
What Success Looks Like in the First 90 Days
  • Can independently pick up a small transform bug or enhancement ticket, make the change, test it, and open a merge request.
  • Comfortable running existing operational scripts to check pipeline status, query tables, and diagnose failures without hand-holding.
  • Starting to take ownership of the Databricks environments and help write new transforms.

$110,000 - $160,000 a year

About Electric Mind

Electric Mind is a fast-growing, AI-native advisory and digital engineering firm built for those who want to shape the future, not just watch it happen. We blend premium strategy expertise with cutting-edge AI-centric engineering to solve complex, meaningful problems for industry-leading clients.

We pride ourselves on a high-touch delivery model and a culture that values diverse talent, innovation, and true client partnership - creating an environment where your ideas matter and your impact is visible.

If you’re looking for a place where you can grow fast, collaborate with exceptional teammates, and help build a company scaling its capabilities and global footprint at speed - Electric Mind is the place to ignite your career. The future is bright!

Electric Mind is committed to diversity in the workplace. We are an inclusive employer and welcome and encourage applications from all qualified candidates. Applicants’ needs will be accommodated during our recruitment and selection process so please advise us if you require accommodation.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Consultant/Manager - Data Engineer
Senior Consultant/Manager - Data Engineer

Electric Mind • Toronto

On-site
CAD 100,000 - 160,000
PySpark / Databricks Developer (Junior to Intermediate)
PySpark / Databricks Developer (Junior to Intermediate)

Electric Mind • Toronto

On-site
CAD 80,000 - 120,000
Databricks Data Platform Engineer
Databricks Data Platform Engineer

Mindtris • Canada

Remote
CAD 120,000 - 180,000
Data Strategy, Advisory & Architecture - Associate Director
Data Strategy, Advisory & Architecture - Associate Director

Electric Mind • Toronto

On-site
CAD 170,000 - 230,000
Data Strategy, Advisory & Architecture - Associate Director
Data Strategy, Advisory & Architecture - Associate Director

Electric Mind • Toronto

On-site
CAD 170,000 - 220,000
Manager, Engineering - AI/BI
Manager, Engineering - AI/BI

Neon • Vancouver

On-site
CAD 171,100 - 227,500
Databricks Developer
Databricks Developer

BeachHead • Toronto

On-site
CAD 103,000 - 106,000
Senior Databricks Data Engineer
Senior Databricks Data Engineer

Raise - find a more meaningful working experience • Toronto

Hybrid
CAD 100,000 - 167,000
Data Engineer - Databricks (Remote)
Data Engineer - Databricks (Remote)

Lumenalta • Winnipeg

Remote
CAD 90,000 - 115,000
Flexible working hours
Health insurance for T4 Employees
Professional development fund
+2
Solutions Architect - MFG - West
Solutions Architect - MFG - West

Databricks Inc. • Vancouver

On-site
CAD 185,000 - 225,000