Senior Data Platform Architect, Cross-Cloud & AI

Quantiphi; Atlanta (GA); On-site; USD 150,000 - 210,000; Full time

Benefits offered by this job

Fortune 500 exposure

Cutting-edge AI tech

Upskilling opportunities

Patent-driven culture

Job summary

Quantiphi, an AI-first digital engineering company, seeks a Technical Architect in Atlanta to design and scale data pipelines, data platforms, and analytics on cloud-native infrastructure. You’ll collaborate with data scientists, analysts, ML engineers, and business stakeholders to deliver robust, scalable solutions.

Role emphasizes cross-cloud migration, large-scale data lake replatforming, and governance using Unity Catalog, with exposure to Databricks, DLT, and GCS.

Qualifications

• Proficient in Databricks on GCP and Azure Databricks.
• Experience with Unity Catalog, Delta Live Tables (DLT) and Auto Loader.
• Familiar with GCP Storage Transfer Service and Google Cloud Storage.
• Strong Python, PySpark, Spark, SQL.
• Experience with Kafka and Pub/Sub.
• Experience with cross-cloud data migrations.

Responsibilities

• Design and build large-scale cross-cloud Data Lake replatforming and migrations from Azure Databricks to GCP Databricks.
• Develop, test, and break-fix batch and streaming ingestion pipelines using Databricks, DLT, and Auto Loader.
• Ingest data from SQL Server, PostgreSQL, Kafka and streaming services (Azure Event Hubs to GCP Pub/Sub).
• Implement pipeline enhancements, path rewrites, secret scope mounting, and incremental state sync.
• Load data and ensure integrity: bulk transfers from Azure ADLS Gen2 to GCS using STS and Databricks Replicator.
• Configure Unity Catalog metastores, schemas, external storage locations, volumes, and RBAC governance across GCP environments.
• Create reusable reporting views in the Silver data layer for downstream metrics.
• Move data from raw to cleaned/transformed layers to align critical business metrics.
• Apply transformations and data mapping for final consumption.
• Implement data quality checks, reconciliation, and pipeline monitoring.
• Optimize data platforms performance and storage structures; set up alerts.
• Enforce data retention/archival policies and validate metric parity.

Tools

Databricks GCP

Azure Databricks

Unity Catalog

Delta Live Tables

Auto Loader

Storage Transfer Service

Google Cloud Storage

Python

PySpark

Spark

Kafka

SQL

Pub/Sub

Cross-Cloud Migration

Job description

Quantiphi, an AI-first digital engineering company, seeks a Technical Architect in Atlanta to design and scale data pipelines, data platforms, and analytics on cloud-native infrastructure. You’ll collaborate with data scientists, analysts, ML engineers, and business stakeholders to deliver robust, scalable solutions.

Role emphasizes cross-cloud migration, large-scale data lake replatforming, and governance using Unity Catalog, with exposure to Databricks, DLT, and GCS.