Contract

CTI Consulting; Quincy (MA); On-site; USD 140,000 - 190,000; Full time

Job summary

CTI Staffing, a leading partner in the Greater Boston area, seeks a Senior Databricks Data Engineer to join the enterprise data team. The role focuses on building a modern Azure data platform that ingests data from diverse sources, enabling reliable, incremental pipelines and faster business decisions.

The ideal candidate will design and optimize end-to-end data pipelines, leverage Delta Lake and Kafka, and collaborate in an Agile environment with developers, architects, and product owners.

Qualifications

• Extensive hands-on Databricks experience with day-one contribution.
• Proven end-to-end data pipelines with CDC.
• Strong PySpark and SQL skills.
• Azure experience with Delta Lake and Kafka.

Responsibilities

• Design, build, and optimize ETL/ELT pipelines using Databricks and PySpark.
• Build CDC pipelines for incremental and near real-time processing.
• Develop scalable Azure data solutions with Delta Lake and Kafka streaming.
• Improve existing pipelines by learning current codebases and upgrading.
• Ingest data from flat files, XML, JSON, SQL Server, JDBC sources, APIs, queues, and Kafka.
• Implement data quality rules and quarantine error records.
• Apply SCD Type 1 and Type 2 modeling.
• Automate workflows with Databricks Workflows and GitHub CI/CD in Agile team.

Skills

Databricks

PySpark

SQL

Azure

Delta Lake

Kafka

CDC

GitHub CI/CD

Tools

GitHub

CI/CD pipelines

Job description

Quincy, United States

CTI Staffing is partnering with a leading retail organization to find a Senior Databricks Data Engineer for their enterprise data team based in the Greater Boston, MA area.

The team is building out a modern Azure data platform that brings together data from many retail systems. This role matters because reliable, incremental pipelines are what let the business trust its data and act on it faster.

What You’ll Do

• Design, build, and optimize ETL/ELT pipelines using Databricks and PySpark.
• Build Change Data Capture pipelines for incremental and near real-time processing.
• Develop scalable data solutions on Azure using Delta Lake and Kafka streaming.
• Improve existing pipelines by quickly learning current codebases and delivering targeted upgrades.
• Ingest and process data from flat files, XML, JSON, SQL Server, JDBC sources, APIs, message queues, and Kafka topics.
• Implement data quality rules and quarantine error records.
• Apply data modeling techniques including SCD Type 1 and Type 2.
• Automate workflows with Databricks Workflows and GitHub-based CI/CD, and work with developers, architects, and product owners in an Agile team.

Requirements

What You Bring

Must-Have:

• Extensive hands-on Databricks experience, able to contribute from day one.
• Proven experience building end-to-end data pipelines with CDC.
• Strong PySpark and SQL skills.
• Experience working in Microsoft Azure.
• Working knowledge of Delta Lake and distributed data processing.
• Experience with Kafka or comparable streaming technologies.
• Hands-on experience with SCD Type 1 and Type 2 data modeling.
• Familiarity with GitHub and CI/CD for version control and deployment.

Nice-to-Have:

• Python experience beyond PySpark.
• Databricks Workflows and Unity Catalog experience.
• Experience with message queues and API-based ingestion.
• Retail, CPG, or supply chain data background.
• Databricks certification.

Technical Environment:

Azure, Databricks, PySpark, SQL, Delta Lake, Kafka, Unity Catalog, GitHub, CI/CD, SQL Server, JSON, XML, REST APIs.