Senior Databricks Data Engineer

Ian Martin Group; Toronto; Hybrid; CAD 100,000 - 167,000; Full time

Job summary

The Ian Martin Group is seeking a Senior Databricks Data Engineer for a Toronto-based hybrid role. You will migrate and refactor pipelines into Databricks, build production-grade data workflows with Python, PySpark and SQL, and translate legacy logic into scalable Databricks-native solutions.

You will integrate data from multiple sources, implement data quality checks, and participate in Agile delivery with client engineering teams.

Qualifications

• 7+ years of data engineering or related experience.
• Strong hands-on Databricks experience.
• Proficiency in Python, PySpark and SQL.
• Experience with ETL/ELT design and large-scale pipelines.
• Experience with Airflow or similar workflow tools.
• Ability to work in Agile/Scrum environments and with client teams.

Responsibilities

• Migrate and refactor ETL/ELT pipelines to Databricks.
• Develop production-grade data pipelines with Python, PySpark, SQL.
• Translate processing logic into scalable Databricks-native solutions.
• Ensure data ingestion, transformation and metrics across sources.
• Modernize workflows and orchestration, including Airflow-based jobs.
• Implement reusable patterns to accelerate migration of workloads.
• Collaborate in Agile delivery two-week sprints with clients.

Skills

Data engineering

Databricks development

Cloud data platforms

Agile/Scrum delivery

Communication

Tools

Databricks

Python

PySpark

SQL

Airflow

Git / CI/CD

Job description

Senior Databricks Data Engineer

• Location: Toronto (Hybrid - minimum 2 days per week onsite at the client's downtown Toronto office)
• Contract Length: 2 Months

Key Responsibilities

Databricks Engineering and Migration

• Migrate and refactor existing ETL/ELT pipelines, scripts and processing workloads into Databricks.
• Develop production-quality data pipelines using Python, PySpark and SQL.
• Translate existing processing and business logic into scalable Databricks-native implementations.
• Develop and maintain ingestion, transformation and metric-calculation logic across multiple enterprise data sources.
• Modernize existing workflow and orchestration processes, including scheduled jobs and Airflow/DAG-based processing.
• Implement reusable engineering patterns to accelerate migration of additional workloads.

Data Integration

• Integrate data from enterprise applications, APIs, databases, files and other source systems.
• Work with source-system SMEs to understand existing data structures, interfaces and processing logic.
• Implement source-to-target mappings based on approved requirements and target data models.
• Support integration with enterprise platforms such as ServiceNow and cybersecurity data sources.
• Troubleshoot connectivity, ingestion, schema and data-quality issues.

Data Quality, Testing and Reconciliation

• Develop automated developer tests and data-validation routines.
• Validate source-to-target transformations and data movement through bronze, silver and gold processing layers where applicable.
• Perform reconciliation between legacy processing outputs and Databricks-generated results.
• Identify, document and resolve data defects, processing exceptions and reconciliation discrepancies.
• Produce clear validation evidence to support business and technical acceptance.

Engineering Quality and DevOps

• Develop maintainable, reusable and well-documented production code.
• Follow enterprise coding, security, architecture and release-management standards.
• Use Git-based repositories and established CI/CD and deployment processes.
• Participate in peer reviews and contribute to improving engineering standards and reusable patterns.
• Support code check-in, deployment readiness, defect remediation and production-transition activities.
• Monitor and troubleshoot pipeline execution and processing failures.

Agile Delivery and Collaboration

• Work within an Agile delivery team executing against two-week sprint commitments.
• Participate in sprint planning, backlog refinement, technical design sessions, defect triage and sprint reviews.
• Estimate engineering effort and identify technical dependencies, blockers and risks.
• Work closely with architects and technical leads to ensure alignment with enterprise architecture and Databricks standards.
• Collaborate directly with client engineering teams and transfer knowledge throughout the engagement.
• Produce technical documentation, source-to-target mappings, implementation notes and handover materials.

Required Skills and Experience

• 7+ years of experience in data engineering, software engineering or data-platform development.
• Strong hands-on experience building production workloads on Databricks.
• Advanced development skills in:
• Python
• PySpark / Apache Spark
• SQL
• Experience designing and implementing ETL/ELT and large-scale data-processing pipelines.
• Experience migrating legacy data pipelines or processing applications to modern cloud data platforms.
• Strong understanding of data ingestion, transformation, modelling, source-to-target mapping, data quality and reconciliation.
• Experience with workflow orchestration technologies such as Apache Airflow or equivalent.
• Experience integrating data through APIs, databases, files and enterprise applications.
• Experience with Git, CI/CD and enterprise software-development practices.
• Strong troubleshooting skills across data, code, integration and environment issues.
• Ability to understand existing code and processing logic and efficiently refactor it into a modern architecture.
• Experience working within structured Agile/Scrum delivery environments.
• Strong written and verbal communication skills with the ability to work directly with client technical teams.
• Must be based in the Greater Toronto Area and available to work onsite in downtown Toronto a minimum of two days per week.

Preferred Qualifications

• Databricks certification such as Databricks Certified Data Engineer Associate or Professional.
• Experience delivering data solutions within financial services, banking or another highly regulated industry.
• Experience with cybersecurity, IT controls, compliance or operational-risk datasets.
• Experience integrating platforms such as ServiceNow, CrowdStrike, BeyondTrust, FireEye, CyberArk, Tenable or similar enterprise security platforms.
• Experience with cloud platforms such as Azure or AWS.
• Experience modernizing Airflow DAGs or other legacy orchestration frameworks.
• Familiarity with enterprise data-governance, metadata, lineage and auditability requirements.
• Experience working in environments with strict security, architecture and release-governance processes.