Senior Data Engineer - Databricks & Data Pipeline Expert

PlanIT Group, LLC; Raleigh (NC); On-site; USD 120,000 - 160,000; Full time

Job summary

PlanIT Group, LLC is seeking a Senior Data Engineer to support data ingestion, de-duplication, and tagging for migrating a large-scale data environment into Databricks. The role focuses on end-to-end data pipeline management, data ingestion from diverse sources, and governance-enabled analytics workflows.

The candidate will work with Azure/Databricks, implement deduplication strategies, and develop tagging frameworks to support metadata, lineage, and compliance.

Qualifications

• 5+ years designing and implementing data ingestion pipelines using Azure Data Factory, Kafka, NiFi, Spark or equivalents.

Responsibilities

• Design, develop, and maintain scalable data ingestion pipelines for batch and streaming data into Azure/Databricks.

Skills

Data pipeline design

Python or scripting

Azure Cloud

Databricks / Spark

Data deduplication

Data tagging / metadata

Communication to CIO

Education

Bachelor's degree in a relevant field

Tools

Azure Data Factory

Apache Kafka

Apache NiFi

Databricks

Apache Spark

Job description

PlanIT Group, LLC is seeking a Senior Data Engineer to support data ingestion, de-duplication, and tagging for migrating a large-scale data environment into Databricks. The role focuses on end-to-end data pipeline management, data ingestion from diverse sources, and governance-enabled analytics workflows.

The candidate will work with Azure/Databricks, implement deduplication strategies, and develop tagging frameworks to support metadata, lineage, and compliance.