Databricks Architect: Real-Time Data & Spark Pipelines

Acunor; Canada; On-site; CAD 100,000 - 130,000; Full time

Job summary

A technology solutions company in Canada is seeking an experienced Databricks Architect to build and optimize ETL pipelines on AWS or Azure. The ideal candidate will have strong expertise in Databricks, Spark, SQL, and Python. Responsibilities include implementing CI/CD pipelines, integrating data using APIs, and leading data engineering initiatives. The role offers a chance to work on large-scale data processing projects within a collaborative environment.

Qualifications

• Strong experience in Databricks and Spark.
• Proficient in SQL, Python, and ETL pipeline development.
• Experience working with AWS or Azure cloud platforms.
• Knowledge of Delta Lake for data processing.

Responsibilities

• Build and optimize ETL pipelines using Databricks and Spark.
• Implement CI/CD pipelines using Azure DevOps.
• Integrate data using APIs and support real-time streaming solutions.
• Lead data engineering projects and collaborate with teams.

Skills

Databricks

Spark

SQL

Python

ETL pipeline development

AWS

Azure

Delta Lake

Tools

Azure DevOps

Airflow

Job description

A technology solutions company in Canada is seeking an experienced Databricks Architect to build and optimize ETL pipelines on AWS or Azure. The ideal candidate will have strong expertise in Databricks, Spark, SQL, and Python. Responsibilities include implementing CI/CD pipelines, integrating data using APIs, and leading data engineering initiatives. The role offers a chance to work on large-scale data processing projects within a collaborative environment.