Senior Data Engineer: Databricks & Data Pipelines Lead

Veriipro; Atlanta (GA); On-site; USD 140,000 - 180,000; Full time

Job summary

Veriipro is seeking an experienced Senior Data Engineer to design and scale ETL pipelines using Databricks, Python, PySpark, and Spark. You will lead data integration efforts, govern data quality, and collaborate with analytics and BI teams on large-scale cloud platforms.

The role requires deep experience with Delta Lake, Unity Catalog, and streaming/batch processing in AWS/GCP environments, along with Agile practices.

Qualifications

• Senior-level data engineering with cloud & big data stack.
• Hands-on experience with Databricks, Delta Lake and Unity Catalog.
• Ability to design scalable ETL pipelines and ensure data governance.

Responsibilities

• Design, develop, and maintain scalable data pipelines and ETL solutions.
• Build data ingestion and transformation frameworks for structured and unstructured data.
• Develop and optimize batch and streaming data processing solutions.
• Implement data quality, validation, and governance standards.
• Integrate heterogeneous data sources into enterprise data platforms.
• Support analytics, reporting, BI, and ML initiatives.
• Collaborate with Data Scientists, BI teams, and business stakeholders.
• Lead technical discussions, mentor junior engineers, and drive data engineering best practices.

Skills

Databricks

Python

PySpark

Apache Spark

ETL development

Data pipelines

Kafka

Hive

HDFS

Sqoop

SQL

Data modeling

Data warehousing

Spark Streaming

Agile methodologies

Tools

Delta Lake

Delta Live Tables

Unity Catalog

AWS

GCP

Tableau

Power BI

Spotfire

OAC

Git

Job description

Veriipro is seeking an experienced Senior Data Engineer to design and scale ETL pipelines using Databricks, Python, PySpark, and Spark. You will lead data integration efforts, govern data quality, and collaborate with analytics and BI teams on large-scale cloud platforms.

The role requires deep experience with Delta Lake, Unity Catalog, and streaming/batch processing in AWS/GCP environments, along with Agile practices.