Consultant - PySpark / Databricks Developer (Junior to Intermediate)

Electric Mind Inc.; Toronto; On-site; CAD 70,000 - 110,000; Full time

Job summary

Electric Mind Inc. is seeking a Consultant - PySpark / Databricks Developer (Junior to Intermediate) to build and maintain data pipelines on Databricks for a financial services data platform.

You’ll work with a senior engineer, writing transforms, fixing pipeline issues, and keeping the environment healthy across dev/qa/uat. The role targets someone with solid Python fundamentals and some Spark/SQL exposure who wants to grow into a data engineering specialist.

Qualifications

• Solid Python fundamentals with clean, readable code.
• Some exposure to Apache Spark / PySpark or strong SQL skills with willingness to learn Spark quickly.
• Working knowledge of SQL (joins, aggregations, window functions).
• Basic Git workflow: branches, commits, pull/merge requests.
• Comfortable reading others' code and debugging methodically.

Responsibilities

• Write and maintain PySpark transforms for bronze/silver/gold data models.
• Work with Databricks' pipeline framework and CDC/ SCD concepts.
• Query and validate data in Unity Catalog across environments using SQL warehouses.
• Debug failed pipeline runs and fix schema drift or data quality issues.
• Maintain reference tables and small utility scripts (PowerShell/Python).
• Write unit/integration tests and participate in code reviews.

Skills

Python

Spark/PySpark exposure

SQL

Git workflow

Debugging

Tools

Databricks

Delta Lake

Azure

Kafka

GitLab CI

Job description

Consultant - PySpark / Databricks Developer (Junior to Intermediate)

AtElectric Mind,Engineeringis where strategy meets action. Our team helps organizations cut through complexity—aligning business ambition with technology execution to unlock real, lasting change.You’llwork alongside curious, driven people tackling high-impact challenges for everyone from scaling startups to global enterprises. Each engagement is different, pushing you to learn, adapt, and grow.

Electric Mind’s Technology Practice brings together deep engineeringexpertise, modern delivery disciplines, and pragmatic architectural thinking to help clients execute complex, mission-critical transformation. We design and implement scalable, secure, high-impact technology solutions that accelerate business outcomes.

About the Role

We are looking for both permeant and contract for this role.
We're looking for a developer to help build and maintain data pipelines on Databricks for a financial services / wealth management data platform. The platform ingests data from multiple source systems, transforms it through a medallion architecture (bronze silver gold) using PySpark, and publishes curated data to downstream consumers via file exports and Kafka. You'll work alongside a senior engineer, writing transform code, fixing pipeline issues, and helping keep the environment healthy across dev/qa/uat.

This is a good fit for someone with solid Python fundamentals and some Spark/SQL exposure who wants to grow into a data engineering specialist. You won't be expected to know Databricks internals on day one — you will be expected to learn fast, ask good questions, and write clean,

What You'll Work On

• Writing and maintainingPySpark transforms(hand-written, not a generic framework) for silver and gold layer tables — things like customer, account, and transaction data models.
• Working withLakeflow Declarative Pipelines(Databricks' current pipeline framework — successor to DLT) andAuto CDC / SCD Type 2patterns for change-data-capture merges.
• Querying and validating data inUnity Catalogacross environments (dev/qa/uat) using SQL warehouses.
• Debugging failed pipeline runs: reading pipeline event logs, tracing bad records, fixing schema drift or data quality issues.
• Maintaining reference/lookup tables and small utility scripts (PowerShell/Python) used to operate the platform.
• Writing and updating unit/integration tests for transform logic.
• Participating in code review, using Git feature branches and merge requests (GitLab).
• Keeping documentation current when you change how something works.

Must-Have Skills

• Python— comfortable writing clean, readable code; understands functions, modules, basic OOP.
• Some exposure toApache Spark / PySpark, or strong SQL skills plus a willingness to learn Spark quickly.
• Working knowledge ofSQL(joins, aggregations, window functions).
• BasicGitworkflow: branches, commits, pull/merge requests.
• Comfortable reading other people's code and stack traces, and debugging methodically (not guess-and-check).

Nice-to-Have Skills

• Direct experience withDatabricks(notebooks, jobs, clusters, or SQL warehouses).
• Familiarity withDelta Lake, medallion architecture (bronze/silver/gold), or CDC/SCD concepts.
• Exposure toAzure(this environment runs on Azure Databricks with Azure AD service principal auth).
• Experience withKafkaor other streaming/event systems.
• Experience withCI/CDpipelines (this project uses GitLab CI).
• Prior work infinancial servicesor a regulated data environment.

What Success Looks Like in the First 90 Days

• Can independently pick up a small transform bug or enhancement ticket, make the change, test it, and open a merge request.
• Comfortable running existing operational scripts to check pipeline status, query tables, and diagnose failures without hand-holding.
• Starting to take ownership of the Databricks environments and help write new transforms