Senior Data Engineer

Veriipro; Atlanta (GA); Vor Ort; USD 140.000 - 180.000; Vollzeit

Zusammenfassung

Veriipro is seeking an experienced Senior Data Engineer to design and scale ETL pipelines using Databricks, Python, PySpark, and Spark. You will lead data integration efforts, govern data quality, and collaborate with analytics and BI teams on large-scale cloud platforms.

The role requires deep experience with Delta Lake, Unity Catalog, and streaming/batch processing in AWS/GCP environments, along with Agile practices.

Qualifikationen

• Senior-level data engineering with cloud & big data stack.
• Hands-on experience with Databricks, Delta Lake and Unity Catalog.
• Ability to design scalable ETL pipelines and ensure data governance.

Aufgaben

• Design, develop, and maintain scalable data pipelines and ETL solutions.
• Build data ingestion and transformation frameworks for structured and unstructured data.
• Develop and optimize batch and streaming data processing solutions.
• Implement data quality, validation, and governance standards.
• Integrate heterogeneous data sources into enterprise data platforms.
• Support analytics, reporting, BI, and ML initiatives.
• Collaborate with Data Scientists, BI teams, and business stakeholders.
• Lead technical discussions, mentor junior engineers, and drive data engineering best practices.

Kenntnisse

Databricks

Python

PySpark

Apache Spark

ETL development

Data pipelines

Kafka

Hive

HDFS

Sqoop

SQL

Data modeling

Data warehousing

Spark Streaming

Agile methodologies

Tools

Delta Lake

Delta Live Tables

Unity Catalog

AWS

GCP

Tableau

Power BI

Spotfire

OAC

Git

Jobbeschreibung

We are looking for an experienced Senior Data Engineer with strong expertise in Databricks, Python, PySpark, Apache Spark, and large-scale data engineering. The ideal candidate will be responsible for designing and developing scalable ETL pipelines, implementing data processing solutions, managing data ingestion frameworks, and optimizing cloud-based data platforms. The role also involves technical leadership, data governance, and collaboration with analytics and business intelligence teams.

Required Technical Skills

• Strong hands-on experience with Databricks, Delta Lake, Delta Live Tables, and Unity Catalog.
• Proficiency in Python, PySpark, Spark SQL, and Apache Spark.
• Strong experience in ETL development, data pipeline design, and data transformation.
• Knowledge of Hadoop, Kafka, Hive, HDFS, and Sqoop.
• Expertise in SQL, data modeling, and data warehousing.
• Experience with batch and real-time data processing using Kafka and Spark Streaming.
• Cloud experience with AWS and/or GCP.
• Strong understanding of data quality, validation, cleansing, and governance.
• Experience working in Agile environments (Scrum, Kanban, SAFe).

Preferred Skills

• Scala / Scala-Spark, IBM DataStage, TIBCO.
• AWS S3, Glue, Lambda, Kinesis/MSK.
• Tableau, Power BI, Spotfire, OAC.
• Teradata, DB2, BigQuery, Mainframe.
• Git, data governance, and retention modeling.

Roles & Responsibilities

• Design, develop, and maintain scalable data pipelines and ETL solutions.
• Build data ingestion and transformation frameworks for structured and unstructured data.
• Develop and optimize batch and streaming data processing solutions.
• Implement data quality, validation, and governance standards.
• Integrate heterogeneous data sources into enterprise data platforms.
• Support analytics, reporting, BI, and machine learning initiatives.
• Collaborate with Data Scientists, BI teams, and business stakeholders.
• Lead technical discussions, mentor junior engineers, and drive data engineering best practices.