Databricks Developer

BeachHead; Toronto; On-site; CAD 103,000 - 106,000; Full time

Job summary

BeachHead Inc. is seeking a hands-on Databricks Developer to design, build, and optimize enterprise-grade data pipelines using Databricks and Apache Spark.

The contract role (6 months) involves large-scale datasets, solution design collaboration, and stakeholder engagement within a financial services context. Preferred experience includes Azure Databricks/Delta Lake, Auto Loader, Unity Catalog, and Lakehouse Federation.

Qualifications

• Hands-on Databricks development experience in data engineering.
• Strong SQL and Python skills for data pipelines.
• Experience with large-scale data processing in enterprise settings.

Responsibilities

• Design, build, and optimize Databricks data pipelines.
• Develop batch and streaming ETL using Auto Loader and Spark Structured Streaming.
• Manage data catalogs, schemas, and governance with Unity Catalog.

Skills

Databricks

Apache Spark

Python

Stakeholder management

Tools

Auto Loader

Unity Catalog

Delta Lake

Spark Structured Streaming

Job description

Are you a hands-on Databricks Developer ready to build scalable data solutions?

Working with one of our top financial clients, this role calls for a Databricks Developer to design, develop, and optimise enterprise-grade data pipelines using Databricks and Apache Spark. This six-month contract offers the opportunity to work with large-scale datasets, contribute to solution design, collaborate with business and technology stakeholders, and apply advanced data engineering practices in a fast-paced environment. Banking or financial services experience is highly preferred.
Pay rate range (CAD): $75.00/hr - $77.00/hr

Responsibilities

• Design, develop, and optimise data pipelines using Databricks and Apache Spark.
• Build scalable ETL pipelines supporting batch processing, Auto Loader, and Spark Structured Streaming.
• Develop and maintain end-to-end data environments, including catalogs, schemas, tables, materialised views, functions, and volumes.
• Implement slowly changing dimensions using SCD 1 and SCD 2 methodologies.
• Build and support change data capture pipelines.
• Use Unity Catalog to manage data governance and access across data environments.
• Develop Lakehouse Federation solutions and create foreign catalogs for external data sources.
• Apply Databricks partitioning and liquid clustering techniques to improve performance.
• Work with large-scale datasets in distributed computing environments.
• Collaborate with cross-functional teams and business stakeholders to understand requirements and deliver effective solutions.
• Contribute to solution design discussions and provide practical technical recommendations.
• Ensure data quality, performance, scalability, and reliability across delivered solutions.
• Communicate technical concepts clearly to stakeholders at varying levels of technical expertise.

Desired Skill-Set

• Strong hands-on experience developing solutions with Databricks and Apache Spark.
• Experience with Azure Databricks or Delta Lake.
• Proven experience building batch and streaming ETL pipelines using Auto Loader and Spark Structured Streaming.
• Strong hands-on experience with Unity Catalog.
• Experience creating catalogs, schemas, tables, materialised views, functions, and volumes.
• Experience implementing SCD 1 and SCD 2 solutions for dimension tables.
• Experience developing change data capture pipelines.
• Strong experience with Lakehouse Federation and foreign catalogs.
• Strong understanding of Databricks partitioning and liquid clustering.
• Experience working in enterprise environments; banking or financial services experience is preferred.
• Strong communication and stakeholder management skills.
• Ability to work independently and effectively in a fast-paced environment.

Nice to Have

• Experience with Kafka.
• Experience with Python.

BeachHead is an equal opportunity agency and employer. We advocate for our candidates and welcome applicants regardless of race, color, religion, national origin, sex, age, or physical or mental disability. BeachHead or our clients may use technology-enabled tools, including automation and artificial intelligence (AI), to support parts of the recruitment process such as resume screening, application management, and candidate matching. These tools assist our recruiters and our clients, and do not replace human decision-making. This job posting represents a current or anticipated vacancy. The position may be filled at any time, and the posting may be removed without notice once the role has been filled.

Privacy Policy