Databricks Data Architect

SGI; Houston (TX); Hybrid; USD 140,000 - 190,000; Full time

Job summary

SGI is seeking a Databricks Data Architect to design and lead modern cloud data solutions on Databricks and AWS. This hands-on role focuses on Lakehouse architecture, streaming, governance, and cloud optimization.

You will design enterprise Databricks Lakehouse architectures, batch and real-time data pipelines, and drive performance and cost improvements across workloads. Hybrid onsite/remote collaboration offered.

Qualifications

• 8+ years of experience in data engineering, data architecture, or cloud data platforms.
• Hands-on Databricks and AWS expertise.
• Strong knowledge of Spark/PySpark and Structured Streaming.
• Experience with Kafka and/or Kinesis/MSK.
• Deep understanding of Delta Lake and Medallion Architecture.
• Familiarity with Unity Catalog, CDC, and data governance.

Responsibilities

• Design enterprise Databricks Lakehouse architectures using Medallion Architecture.
• Architect batch and real-time data solutions with PySpark, Structured Streaming, Kafka, Delta Lake, and AWS.
• Address late-arriving data, event-time processing, watermarking, CDC, CDF, and streaming recovery.
• Leverage Databricks features including Lakeflow, Serverless Compute, Unity Catalog, and Photon.
• Evaluate new capabilities such as Zerobus, Lakebase-to-Lakehouse sync, and LTAP.
• Design and optimize AWS data platforms using S3, Glue, EMR, Kinesis/MSK, Redshift, Lambda, IAM, CloudWatch.
• Drive performance and cost optimization across Databricks and AWS workloads.
• Establish standards for data governance, security, lineage, CI/CD, and infrastructure as code.
• Lead technical design discussions and mentor Data Engineers.

Skills

Databricks

AWS

Spark/PySpark

Structured Streaming

Kafka

Kinesis

Delta Lake

Medallion Architecture

Unity Catalog

Z-Ordering

Serverless Compute

Lakehouse

Data Governance

CI/CD

Infrastructure as Code

Performance Tuning

Cost Optimization

Lakeflow

Lakebase

LTAP

Zerobus

Tools

Lakeflow

Lakebase

LTAP

Zerobus

Job description

Work Arrangement: Hybrid - 3 days onsite per week

Contract: 12 months

We are seeking a Databricks Data Architect to design and lead modern cloud data solutions using Databricks and AWS. This is a hands-on architecture role focused on Lakehouse architecture, streaming, performance, governance, and cloud optimization.

Key Responsibilities

• Design enterprise Databricks Lakehouse architectures using Medallion Architecture.
• Architect batch and real-time data solutions using PySpark, Structured Streaming, Kafka, Delta Lake, and AWS.
• Design solutions for late-arriving data, event-time processing, watermarking, CDC, CDF, and streaming recovery.
• Leverage modern Databricks capabilities including Lakeflow/LDP, Serverless Compute, Unity Catalog, Liquid Clustering, Z-Ordering, and Photon.
• Evaluate emerging capabilities including Zerobus, Lakebase-to-Lakehouse synchronization, and LTAP.
• Design and optimize AWS data platforms using S3, Glue, EMR, Kinesis/MSK, Redshift, Lambda, IAM, and CloudWatch.
• Drive performance and cost optimization across Databricks and AWS workloads.
• Establish standards for data governance, security, lineage, CI/CD, and infrastructure as code.
• Lead technical design discussions and mentor Data Engineers.

Required Experience

• 8+ years of experience in Data Engineering, Data Architecture, or Cloud Data Platforms.
• Strong hands‑on experience with Databricks and AWS.
• Expert knowledge of Spark/PySpark and Structured Streaming.
• Strong experience with Kafka and/or Kinesis/MSK.
• Deep understanding of Delta Lake and Medallion Architecture.
• Experience with Unity Catalog, CDF/CDC, Liquid Clustering, and Z-Ordering.
• Strong understanding of Databricks performance and cost optimization.
• Excellent communication and stakeholder management skills.

Preferred: Databricks certification, AWS certification, and experience with newer Databricks capabilities such as Lakeflow, Zerobus, Lakebase, Serverless Compute, and LTAP.