Senior Databricks Data Engineer - Real-Time Azure Platform

CTI Consulting; Quincy (MA); On-site; USD 140,000 - 190,000; Full time

Job summary

CTI Staffing, a leading partner in the Greater Boston area, seeks a Senior Databricks Data Engineer to join the enterprise data team. The role focuses on building a modern Azure data platform that ingests data from diverse sources, enabling reliable, incremental pipelines and faster business decisions.

The ideal candidate will design and optimize end-to-end data pipelines, leverage Delta Lake and Kafka, and collaborate in an Agile environment with developers, architects, and product owners.

Qualifications

• Extensive hands-on Databricks experience with day-one contribution.
• Proven end-to-end data pipelines with CDC.
• Strong PySpark and SQL skills.
• Azure experience with Delta Lake and Kafka.

Responsibilities

• Design, build, and optimize ETL/ELT pipelines using Databricks and PySpark.
• Build CDC pipelines for incremental and near real-time processing.
• Develop scalable Azure data solutions with Delta Lake and Kafka streaming.
• Improve existing pipelines by learning current codebases and upgrading.
• Ingest data from flat files, XML, JSON, SQL Server, JDBC sources, APIs, queues, and Kafka.
• Implement data quality rules and quarantine error records.
• Apply SCD Type 1 and Type 2 modeling.
• Automate workflows with Databricks Workflows and GitHub CI/CD in Agile team.

Skills

Databricks

PySpark

SQL

Azure

Delta Lake

Kafka

CDC

GitHub CI/CD

Tools

GitHub

CI/CD pipelines

Job description

CTI Staffing, a leading partner in the Greater Boston area, seeks a Senior Databricks Data Engineer to join the enterprise data team. The role focuses on building a modern Azure data platform that ingests data from diverse sources, enabling reliable, incremental pipelines and faster business decisions.

The ideal candidate will design and optimize end-to-end data pipelines, leverage Delta Lake and Kafka, and collaborate in an Agile environment with developers, architects, and product owners.