Senior Data Engineer: Cloud Lakehouse & Real‑Time Pipelines

Quantiphi; Atlanta (GA); On-site; USD 140,000 - 180,000; Full time

Job summary

Quantiphi is seeking a senior data engineer to design, build, and scale robust data pipelines on cloud-native platforms. You will collaborate with data scientists, analysts, and stakeholders to deliver scalable data lakehouse solutions using Databricks across GCP and Azure.

You will write Python and PySpark, manage streaming pipelines with Kafka, and lead cross-cloud migrations while ensuring data quality, governance, and security.

Qualifications

• Bachelor's or Master's degree in CS/IT/Engineering or related field.
• 5+ years of professional data engineering experience.
• Proven track record delivering end-to-end data pipelines in production.
• Strong understanding of data warehousing, dimensional modeling, and data lakehouse architecture.
• Experience in Agile/Scrum environments.
• Excellent problem-solving, communication, and collaboration skills.

Responsibilities

• Design, develop, and maintain scalable data pipelines and ELT/ETL workflows using Databricks (GCP & Azure) and DBT.
• Build and manage Delta Live Tables (DLT) pipelines and leverage Auto Loader for incremental data ingestion.
• Architect data lakehouse solutions using Databricks including Unity Catalog for governance and Delta Lake.
• Develop and maintain data solutions on GCP: GCS, Pub/Sub, STS, BigQuery, Cloud Composer, Dataflow.
• Write production-grade code in Python and PySpark/Spark for distributed processing.
• Build and maintain Kafka-based streaming pipelines for real-time ingestion.
• Develop complex SQL queries for extraction, transformation, aggregation, validation.
• Lead cross-cloud migration initiatives (GCP, Azure, AWS).
• Implement data quality checks, monitoring, and alerting across pipelines.
• Ensure governance, security, and privacy best practices.
• Mentor junior data engineers, conduct code reviews, and promote best practices.

Skills

Databricks on GCP

Azure Databricks

Unity Catalog

Delta Live Tables

Auto Loader

GCS

Python

PySpark / Spark

Kafka

SQL

Pub/Sub

Cross-Cloud Migration

Education

Bachelor's or Master's in CS/IT/Engineering

Tools

DBT

Apache Airflow

Cloud Composer

Terraform

Dataflow

Dataplex

Collibra

Alation

MLOps

BigQuery

Job description

Quantiphi is seeking a senior data engineer to design, build, and scale robust data pipelines on cloud-native platforms. You will collaborate with data scientists, analysts, and stakeholders to deliver scalable data lakehouse solutions using Databricks across GCP and Azure.

You will write Python and PySpark, manage streaming pipelines with Kafka, and lead cross-cloud migrations while ensuring data quality, governance, and security.