Databricks Architect

Acunor; Canada; On-site; CAD 100,000 - 130,000; Full time

Job summary

A technology solutions company in Canada is seeking an experienced Databricks Architect to build and optimize ETL pipelines on AWS or Azure. The ideal candidate will have strong expertise in Databricks, Spark, SQL, and Python. Responsibilities include implementing CI/CD pipelines, integrating data using APIs, and leading data engineering initiatives. The role offers a chance to work on large-scale data processing projects within a collaborative environment.

Qualifications

• Strong experience in Databricks and Spark.
• Proficient in SQL, Python, and ETL pipeline development.
• Experience working with AWS or Azure cloud platforms.
• Knowledge of Delta Lake for data processing.

Responsibilities

• Build and optimize ETL pipelines using Databricks and Spark.
• Implement CI/CD pipelines using Azure DevOps.
• Integrate data using APIs and support real-time streaming solutions.
• Lead data engineering projects and collaborate with teams.

Skills

Databricks

Spark

SQL

Python

ETL pipeline development

AWS

Azure

Delta Lake

Tools

Azure DevOps

Airflow

Job description

We are looking for a Databricks Architect with strong experience in building and optimizing ETL pipelines on cloud platforms like AWS or Azure. This role involves working on large scale data processing real time pipelines and leading data engineering initiatives.

Build and optimize ETL pipelines using Databricks and Spark

Work with SQL Python PySpark and Delta Lake for data processing

Implement CI CD pipelines using Azure DevOps and manage workflows with Airflow

Integrate data using APIs and support real time streaming solutions

Lead data engineering projects and collaborate with cross functional teams

Must Have Skills

• Strong Databricks and Spark experience
• SQL, Python and ETL pipeline development
• Experience with cloud platforms AWS or Azure
• Experience with Delta Lake and data processing