Data Engineer - PySpark & Databricks Pipelines

Electric Mind Inc.; Toronto (SD); On-site; USD 100,000 - 140,000; Full time

Job summary

Electric Mind is seeking a data engineer to build and maintain data pipelines on Databricks for a financial services data platform. You will work on a medallion (bronze/silver/gold) architecture using PySpark and publish curated data to downstream consumers via file exports and Kafka.

You’ll collaborate with a senior engineer, fix pipeline issues, and help keep the environment healthy across dev/qa/uat, with a focus on learning fast and writing clean, reliable code.

Qualifications

• Proficient in Python with clean, readable code and basic OOP.
• Experience with Databricks, notebooks, jobs, clusters, or SQL warehouses is a plus.
• Exposure to Spark/PySpark or strong SQL skills and willingness to learn Spark quickly.
• Working knowledge of SQL (joins, aggregations, window functions).
• Basic Git workflow: branches, commits, pull/merge requests.
• Able to read other people’s code and debug methodically.

Responsibilities

• Write and maintain PySpark transforms for bronze/silver/gold tables.
• Work with Lakeflow Declarative Pipelines and CDC/SCD patterns.
• Query and validate data in Unity Catalog across environments using SQL warehouses.
• Debug failed pipeline runs by tracing logs and fixing data quality issues.
• Maintain reference/lookup tables and small utility scripts (PowerShell/Python).
• Write and update unit/integration tests for transform logic.
• Participate in code reviews using GitLab MR workflows.
• Keep documentation current when changes are made.

Skills

Python

Databricks

Spark/PySpark exposure

SQL

Git workflow

Debugging

Tools

Databricks

Apache Spark

Job description

Electric Mind is seeking a data engineer to build and maintain data pipelines on Databricks for a financial services data platform. You will work on a medallion (bronze/silver/gold) architecture using PySpark and publish curated data to downstream consumers via file exports and Kafka.

You’ll collaborate with a senior engineer, fix pipeline issues, and help keep the environment healthy across dev/qa/uat, with a focus on learning fast and writing clean, reliable code.