Senior Data Engineer

Quantiphi; Atlanta (GA); On-site; USD 140,000 - 180,000; Full time

Job summary

Quantiphi is seeking a senior data engineer to design, build, and scale robust data pipelines on cloud-native platforms. You will collaborate with data scientists, analysts, and stakeholders to deliver scalable data lakehouse solutions using Databricks across GCP and Azure.

You will write Python and PySpark, manage streaming pipelines with Kafka, and lead cross-cloud migrations while ensuring data quality, governance, and security.

Qualifications

• Bachelor's or Master's degree in CS/IT/Engineering or related field.
• 5+ years of professional data engineering experience.
• Proven track record delivering end-to-end data pipelines in production.
• Strong understanding of data warehousing, dimensional modeling, and data lakehouse architecture.
• Experience in Agile/Scrum environments.
• Excellent problem-solving, communication, and collaboration skills.

Responsibilities

• Design, develop, and maintain scalable data pipelines and ELT/ETL workflows using Databricks (GCP & Azure) and DBT.
• Build and manage Delta Live Tables (DLT) pipelines and leverage Auto Loader for incremental data ingestion.
• Architect data lakehouse solutions using Databricks including Unity Catalog for governance and Delta Lake.
• Develop and maintain data solutions on GCP: GCS, Pub/Sub, STS, BigQuery, Cloud Composer, Dataflow.
• Write production-grade code in Python and PySpark/Spark for distributed processing.
• Build and maintain Kafka-based streaming pipelines for real-time ingestion.
• Develop complex SQL queries for extraction, transformation, aggregation, validation.
• Lead cross-cloud migration initiatives (GCP, Azure, AWS).
• Implement data quality checks, monitoring, and alerting across pipelines.
• Ensure governance, security, and privacy best practices.
• Mentor junior data engineers, conduct code reviews, and promote best practices.

Skills

Databricks on GCP

Azure Databricks

Unity Catalog

Delta Live Tables

Auto Loader

GCS

Python

PySpark / Spark

Kafka

SQL

Pub/Sub

Cross-Cloud Migration

Education

Bachelor's or Master's in CS/IT/Engineering

Tools

DBT

Apache Airflow

Cloud Composer

Terraform

Dataflow

Dataplex

Collibra

Alation

MLOps

BigQuery

Job description

Quantiphi is an award-winning, AI-First global digital engineering company that helps the world’s leading Fortune 1000 organizations transform bold ideas into measurable business impact. We go beyond building innovative AI technologies—we solve the problems that matter most to our clients.

Since our founding in 2013, Quantiphi has built a proven track record of turning complex challenges into meaningful outcomes across industries.

Headquartered in Boston, with more than 4,000 professionals worldwide, we partner with global enterprises to deliver large-scale digital, cloud, and AI-driven transformation. #SolvingWhatMatters

We are an Elite and Premier partner to Google Cloud, AWS, NVIDIA, Snowflake, and other leading technology platforms, and our work has been recognized across the industry, including:

• 21 Google Cloud Partner of the Year awards in the past 10 years
• 3 AWS AI/ML Partner of the Year awards
• 3 NVIDIA Partner of the Year awards
• 3 Snowflake Partner of the Year awards
• Rated Leaders by Gartner, Forrester, IDC, ISG, Everest Group and other leading analyst firms

Quantiphi delivers First-in-class AI solutions across Life Sciences, Healthcare, Banking, Financial Services, CPG, Manufacturing, Energy, High-Tech, Telecommunications, etc., powered by cutting-edge Generative AI and Agentic AI accelerators.

We are also proud to be certified as a Great Place to Work—reflecting our commitment to our people and our culture.

Job Description -

In this role, you will design, build, and scale robust data pipelines, data platforms, and analytical solutions on cloud-native infrastructure, working closely with data scientists, analysts, ML engineers, and business stakeholders.

Key Responsibilities -

• Design, develop, and maintain scalable data pipelines and ELT/ETL workflows using Databricks (GCP & Azure) and DBT
• Build and manage Delta Live Tables (DLT) pipelines and leverage Auto Loader for incremental data ingestion
• Architect and manage data lakehouse solutions using Databricks — including Unity Catalog for data governance and Delta Lake
• Develop and maintain data solutions on Google Cloud Platform (GCP) — GCS, Pub/Sub, GCP Storage Transfer Service (STS), BigQuery, Cloud Composer, and Dataflow
• Write efficient, production-grade code in Python and PySpark/Spark for distributed data processing and automation
• Build and maintain Kafka-based streaming pipelines for real-time data ingestion and processing
• Develop complex SQL queries for data extraction, transformation, aggregation, and validation
• Lead and execute cross-cloud migration initiatives (e.g., migrating workloads between GCP, Azure, AWS)
• Implement data quality checks, monitoring, and alerting across data pipelines
• Ensure compliance with data governance, security, and privacy best practices
• Mentor and guide junior data engineers, conduct code reviews, and champion engineering best practices

Must-Have Skills -

• Databricks on GCP: Hands-on experience with Databricks on Google Cloud Platform — clusters, jobs, workflows, and integrations
• Azure Databricks: Experience deploying and managing Databricks workspaces on Microsoft Azure
• Unity Catalog: Proficiency in configuring and managing Unity Catalog for unified data governance across workspaces
• Delta Live Tables (DLT): Experience building and managing DLT pipelines for declarative ETL on Databricks
• Auto Loader: Hands-on experience with Databricks Auto Loader for scalable, incremental file ingestion
• Google Cloud Storage (GCS): Strong hands-on experience with GCS for data lake storage, lifecycle policies, and integrations
• Python: Strong programming skills for data engineering, scripting, and automation
• PySpark / Spark: Experience building distributed data processing applications using PySpark and Apache Spark
• Kafka: Experience designing and managing Kafka-based real-time streaming pipelines
• SQL: Advanced SQL skills including query optimization, window functions, and complex joins
• Pub/Sub: Hands-on experience with Google Cloud Pub/Sub for event-driven and streaming architectures
• Cross-Cloud Migration: Proven experience migrating data workloads and pipelines across cloud platforms (GCP, Azure, AWS)

Good-to-Have Skills -

• DBT (Data Build Tool) — models, tests, macros, and project management
• Apache Airflow / Cloud Composer for workflow orchestration
• Terraform or other Infrastructure-as-Code (IaC) tools
• Dataflow for batch and streaming data processing
• Data cataloging and governance tools (Dataplex, Collibra, Alation)
• MLOps or feature engineering pipeline support
• BigQuery for large-scale analytics workloads

Qualifications -

• Bachelor's or Master's degree in Computer Science, IT, Engineering, or related field.
• 5+ years of professional experience in data engineering.
• Proven track record of delivering end-to-end data pipeline solutions in production.
• Strong understanding of data warehousing, dimensional modeling, and data lakehouse architecture.
• Experience in Agile/Scrum environments.
• Excellent problem-solving, communication, and collaboration skills.