Data Bricks Engineer

V-Soft Consulting Group, Inc.; Canada; On-site; CAD 95,000 - 130,000; Full time

Job summary

V-Soft Consulting Group, Inc. is seeking a Databricks Data Engineer to design, build, and optimize end-to-end data pipelines in an Azure environment. The role emphasizes PySpark, SQL, and Kafka streaming to deliver reliable data solutions for retail datasets.

You will implement CDC-based pipelines, ensure data quality, and collaborate across teams to move from development to production with CI/CD practices. Familiarity with Delta Lake and data modeling is a plus.

Qualifications

• Hands-on Databricks experience with ability to contribute from day one.
• End-to-end data pipelines with CDC capabilities.
• Strong PySpark and SQL; Python a plus.
• Azure cloud experience and Databricks integration.
• Familiarity with GitHub CI/CD pipelines for deployment.

Responsibilities

• Design, build, and optimize end-to-end ETL/ELT pipelines using Databricks and Spark (PySpark).
• Develop and maintain scalable data solutions on Azure with Databricks, Delta Lake, and Kafka.
• Build robust CDC-enabled pipelines for real-time data processing.
• Enhance existing pipelines and rapidly understand current codebases to deliver improvements.
• Collaborate with architects, developers, and product owners in Agile/Scrum ceremonies.

Skills

Databricks

CDC pipelines

PySpark

SQL

Azure

Kafka

GitHub CI/CD

Data architecture

Performance optimization

Pipeline scalability

Tools

Databricks Platform

Delta Lake

Kafka

Job description

Databricks Data Engineer with Azure, PySpark, Kafka and end-to-end automation of data engineering pipelines.

Self healing pipeline, streaming pipelines, automating pipelines and Databricks Certified

Description:

We are seeking a highly skilled Databricks Data Engineer to support the Retail Spine initiative, working with a wide range of retail datasets to design, build, and optimize scalable data solutions. This role requires deep hands-on expertise in Databricks and the ability to contribute immediately by developing new pipelines while enhancing existing data workflows.

The ideal candidate brings extensive experience building end-to-end data engineering pipelines with CDC (Change Data Capture) capabilities, and thrives in a fast-paced Azure-based environment leveraging Databricks, PySpark, and Kafka streaming.

Key Responsibilities

• Design, build, and optimize end-to-end ETL/ELT pipelines using Databricks and Apache Spark (PySpark).
• Develop and maintain scalable data solutions on Microsoft Azure, leveraging Databricks, Delta Lake, and Kafka streaming.
• Build robust pipelines with Change Data Capture (CDC) to support incremental data processing and real-time/near real-time data needs.
• Work across both new pipeline development and enhancement of existing pipelines, quickly understanding current codebases to deliver improvements.
• Integrate and process data from diverse sources including:
• Flat files, XML, JSON, SQL Server and JDBC systems, Kafka topics, APIs, MQ, Unity Catalog volumes and other enterprise platforms.
• Implement data quality frameworks, including writing technical data quality rules and quarantining error records.
• Apply canonical data modeling techniques, including SCD Type 1 and Type 2 implementations.
• Ensure high levels of data reliability, performance, scalability, and monitoring across all workflows.
• Automate workflows using Databricks Workflows and CI/CD pipelines (GitHub), focusing on reuse, efficiency, and standardization.
• Adhere to ADUSA architecture, governance, and security standards.
• Collaborate with developers, architects, and product owners, and engage with stakeholders, source teams, and downstream consumers as needed.
• Participate actively in Agile/Scrum ceremonies and deliver high-quality solutions within defined timelines.

Required Qualifications

• Extensive hands-on experience with Databricks, with ability to contribute from day one.
• Proven experience building end-to-end data engineering pipelines with CDC capabilities.
• Strong proficiency in PySpark and SQL (required); Python experience is a plus.
• Experience working in Microsoft Azure cloud environments.
• Knowledge of:
• Distributed data processing frameworks
• Streaming technologies (Kafka)
• Data modeling (SCD Type 1 & Type 2)
• Experience integrating data from multiple structured and unstructured data sources.
• Familiarity with GitHub and CI/CD pipelines for version control and deployment.
• Strong understanding of data architecture, performance optimization, and pipeline scalability.