Technical Architect

Quantiphi; Atlanta (GA); On-site; USD 150,000 - 210,000; Full time

Benefits offered by this job

Fortune 500 exposure

Cutting-edge AI tech

Upskilling opportunities

Patent-driven culture

Job summary

Quantiphi, an AI-first digital engineering company, seeks a Technical Architect in Atlanta to design and scale data pipelines, data platforms, and analytics on cloud-native infrastructure. You’ll collaborate with data scientists, analysts, ML engineers, and business stakeholders to deliver robust, scalable solutions.

Role emphasizes cross-cloud migration, large-scale data lake replatforming, and governance using Unity Catalog, with exposure to Databricks, DLT, and GCS.

Qualifications

• Proficient in Databricks on GCP and Azure Databricks.
• Experience with Unity Catalog, Delta Live Tables (DLT) and Auto Loader.
• Familiar with GCP Storage Transfer Service and Google Cloud Storage.
• Strong Python, PySpark, Spark, SQL.
• Experience with Kafka and Pub/Sub.
• Experience with cross-cloud data migrations.

Responsibilities

• Design and build large-scale cross-cloud Data Lake replatforming and migrations from Azure Databricks to GCP Databricks.
• Develop, test, and break-fix batch and streaming ingestion pipelines using Databricks, DLT, and Auto Loader.
• Ingest data from SQL Server, PostgreSQL, Kafka and streaming services (Azure Event Hubs to GCP Pub/Sub).
• Implement pipeline enhancements, path rewrites, secret scope mounting, and incremental state sync.
• Load data and ensure integrity: bulk transfers from Azure ADLS Gen2 to GCS using STS and Databricks Replicator.
• Configure Unity Catalog metastores, schemas, external storage locations, volumes, and RBAC governance across GCP environments.
• Create reusable reporting views in the Silver data layer for downstream metrics.
• Move data from raw to cleaned/transformed layers to align critical business metrics.
• Apply transformations and data mapping for final consumption.
• Implement data quality checks, reconciliation, and pipeline monitoring.
• Optimize data platforms performance and storage structures; set up alerts.
• Enforce data retention/archival policies and validate metric parity.

Tools

Databricks GCP

Azure Databricks

Unity Catalog

Delta Live Tables

Auto Loader

Storage Transfer Service

Google Cloud Storage

Python

PySpark

Spark

Kafka

SQL

Pub/Sub

Cross-Cloud Migration

Job description

Quantiphi is an award-winning, AI-First global digital engineering company that helps the world's leading Fortune 1000 organizations transform bold ideas into measurable business impact. We go beyond building innovative AI technologies—we solve the problems that matter most to our clients.

Since our founding in 2013, Quantiphi has built a proven track record of turning complex challenges into meaningful outcomes across industries.

Headquartered in Boston, with more than 4,000 professionals worldwide, we partner with global enterprises to deliver large-scale digital, cloud, and AI-driven transformation.

#SolvingWhatMatters

We are an Elite and Premier partner to Google Cloud, AWS, NVIDIA, Snowflake, and other leading technology platforms, and our work has been recognized across the industry, including:

• 21 Google Cloud Partner of the Year awards in the past 10 years
• 3 AWS AI/ML Partner of the Year awards
• 3 NVIDIA Partner of the Year awards
• 3 Snowflake Partner of the Year awards
• Rated Leaders by Gartner, Forrester, IDC, ISG, Everest Group and other leading analyst firms

Quantiphi delivers First-in-class AI solutions across Life Sciences, Healthcare, Banking, Financial Services, CPG, Manufacturing, Energy, High-Tech, Telecommunications, etc., powered by cutting-edge Generative AI and Agentic AI accelerators.

We are also proud to be certified as a Great Place to Work—reflecting our commitment to our people and our culture.

Role: Technical Architect

Experience Level: 8+ Years

Employment type: Full Time

Location: Atlanta, Georgia, USA

Job Description:

In this role, you will design, build, and scale robust data pipelines, data platforms, and analytical solutions on cloud-native infrastructure, working closely with data scientists, analysts, ML engineers, and business stakeholders.

Key Responsibilities

• Experience designing and executing large-scale cross-cloud Data Lake replatforming and migrations from Azure Databricks to GCP Databricks.
• Develop, test, and perform break-fix operations for batch and streaming ingestion pipelines (Bronze/Silver/Gold medallion layers) using Databricks, DLT, and Auto Loader.
• Build and refactor pipelines to ingest data from diverse source systems (SQL Server, PostgreSQL, Kafka) and streaming services (converting Azure Event Hubs to GCP Pub/Sub).
• Implement pipeline enhancements, path rewrites (abfss:// to gs://), secret scope mounting via GCP Secret Manager, and incremental state sync logic.
• Data Loading & Integrity:
• Design and execute bulk data transfers and historical backfills from Azure ADLS Gen2 to Google Cloud Storage (GCS) using GCP Storage Transfer Service (STS) and Databricks Replicator.
• Configure central Unity Catalog metastores, schemas, external storage locations, volumes, and RBAC governance across GCP environments.
• Reporting & Metrics Enablement:
• Build reusable reporting views or curated tables in the cleaned/transformed (Silver) data layer for downstream consumption.
• Develop pipelines needed to move data from raw to cleaned/transformed layers to support the alignment of critical business metrics.
• Implement necessary transformations, complex data mapping, and alignment activities to prepare data for final consumption.
• Platform Optimization & Quality:
• Implement data quality checks, reconciliation processes, and pipeline monitoring.
• Contribute to data platform optimization efforts, including performance tuning of data processing jobs, optimizing storage structures, and configuring alerts for failures and delays.
• Enforce data retention/archival policies and validate metric parity between source and target systems.

Required Skills & Qualifications

Technical Skills

• Databricks GCP, Azure Databricks, Unity Catalog, Delta Live Tables (DLT) pipelines, Auto Loader, GCP Storage Transfer Service (STS), Google Cloud Storage (GCS), Python, PySpark, Spark, Kafka, SQL, Pub/Sub, Cross-Cloud Migration.

Preferred Certifications:

What's in it for YOU at Quantiphi?

• Join one of the world's fastest-growing AI-first digital engineering companies and make a real impact at scale.
• Lead and collaborate with a high-energy team of talented, driven individuals solving complex, meaningful challenges.
• Work with Fortune 500 companies and disruptive innovators in a research-driven environment with 60+ patents.
• Stay ahead of the curve by gaining hands-on experience with cutting-edge AI, ML, data, and cloud technologies while continuously upskilling.