Databricks Developer

BeachHead; Toronto; On-site; CAD 103,000 - 106,000; Full time

Job summary

BeachHead is seeking a hands-on Databricks Developer to design, build, and optimise enterprise-grade data pipelines using Databricks and Apache Spark. This six-month contract involves large-scale datasets, collaboration with business and tech stakeholders, and advanced data engineering practices in a fast-paced environment, with banking experience preferred.

The role focuses on scalable ETL pipelines, Unity Catalog governance, and Lakehouse Federation solutions, delivering reliable and efficient

Qualifications

• Extensive hands-on experience with Databricks and Apache Spark in enterprise environments.
• Strong background building batch and streaming ETL pipelines.
• Experience with Unity Catalog and data governance practices.
• Experience configuring catalogs, schemas, tables, materialised views, functions, and volumes.

Responsibilities

• Design, develop, and optimise data pipelines using Databricks and Spark.
• Build scalable ETL pipelines supporting batch processing and Spark Structured Streaming.
• Develop end-to-end data environments including catalogs, schemas, tables, and volumes.
• Implement SCD 1 and SCD 2 for dimension tables and manage CDC pipelines.

Skills

Databricks

Apache Spark

ETL pipelines

Data integration

Streaming

Tools

Azure Databricks

Delta Lake

Unity Catalog

Lakehouse Federation

Job description

Are you a hands-on Databricks Developer ready to build scalable data solutions?

Working with one of our top financial clients, this role calls for a Databricks Developer to design, develop, and optimise enterprise-grade data pipelines using Databricks and Apache Spark. This six-month contract offers the opportunity to work with large-scale datasets, contribute to solution design, collaborate with business and technology stakeholders, and apply advanced data engineering practices in a fast-paced environment. Banking or financial services experience is highly preferred.

Pay rate range (CAD): $75.00/hr – $77.00/hr

Responsibilities

• Design, develop, and optimise data pipelines using Databricks and Apache Spark.
• Build scalable ETL pipelines supporting batch processing, Auto Loader, and Spark Structured Streaming.
• Develop and maintain end-to-end data environments, including catalogs, schemas, tables, materialised views, functions, and volumes.
• Implement slowly changing dimensions using SCD 1 and SCD 2 methodologies.
• Build and support change data capture pipelines.
• Use Unity Catalog to manage data governance and access across data environments.
• Develop Lakehouse Federation solutions and create foreign catalogs for external data sources.
• Apply Databricks partitioning and liquid clustering techniques to improve performance.
• Work with large-scale datasets in distributed computing environments.
• Collaborate with cross-functional teams and business stakeholders to understand requirements and deliver effective solutions.
• Contribute to solution design discussions and provide practical technical recommendations.
• Ensure data quality, performance, scalability, and reliability across delivered solutions.
• Communicate technical concepts clearly to stakeholders at varying levels of technical expertise.

Desired Skill-Set

• Strong hands-on experience developing solutions with Databricks and Apache Spark.
• Experience with Azure Databricks or Delta Lake.
• Proven experience building batch and streaming ETL pipelines using Auto Loader and Spark Structured Streaming.
• Strong hands-on experience with Unity Catalog.
• Experience creating catalogs, schemas, tables, materialised views, functions, and volumes.
• Experience implementing SCD 1 and SCD 2 solutions for dimension tables.
• Experience developing change data capture pipelines.
• Strong experience with Lakehouse Federation and foreign catalogs.
• Strong understanding of Databricks partitioning and liquid clustering.
• Experience working in enterprise environments; banking or financial services experience is preferred.
• Strong communication and stakeholder management skills.
• Ability to work independently and effectively in a fast-paced environment.

Nice to Have

• Experience with Kafka.
• Experience with Python.

BeachHead is an equal opportunity agency and employer. We advocate for our candidates and welcome applicants regardless of race, color, religion, national origin, sex, age, or physical or mental disability. BeachHead or our clients may use technology-enabled tools, including automation and artificial intelligence (AI), to support parts of the recruitment process such as resume screening, application management, and candidate matching. These tools assist our recruiters and our clients, and do not replace human decision-making. This job posting represents a current or anticipated vacancy. The position may be filled at any time, and the posting may be removed without notice once the role has been filled.

Privacy Policy