Hybrid Databricks Lakehouse Architect

SGI; Houston (TX); Hybrid; USD 140,000 - 190,000; Full time

Job summary

SGI is seeking a Databricks Data Architect to design and lead modern cloud data solutions on Databricks and AWS. This hands-on role focuses on Lakehouse architecture, streaming, governance, and cloud optimization.

You will design enterprise Databricks Lakehouse architectures, batch and real-time data pipelines, and drive performance and cost improvements across workloads. Hybrid onsite/remote collaboration offered.

Qualifications

• 8+ years of experience in data engineering, data architecture, or cloud data platforms.
• Hands-on Databricks and AWS expertise.
• Strong knowledge of Spark/PySpark and Structured Streaming.
• Experience with Kafka and/or Kinesis/MSK.
• Deep understanding of Delta Lake and Medallion Architecture.
• Familiarity with Unity Catalog, CDC, and data governance.

Responsibilities

• Design enterprise Databricks Lakehouse architectures using Medallion Architecture.
• Architect batch and real-time data solutions with PySpark, Structured Streaming, Kafka, Delta Lake, and AWS.
• Address late-arriving data, event-time processing, watermarking, CDC, CDF, and streaming recovery.
• Leverage Databricks features including Lakeflow, Serverless Compute, Unity Catalog, and Photon.
• Evaluate new capabilities such as Zerobus, Lakebase-to-Lakehouse sync, and LTAP.
• Design and optimize AWS data platforms using S3, Glue, EMR, Kinesis/MSK, Redshift, Lambda, IAM, CloudWatch.
• Drive performance and cost optimization across Databricks and AWS workloads.
• Establish standards for data governance, security, lineage, CI/CD, and infrastructure as code.
• Lead technical design discussions and mentor Data Engineers.

Skills

Databricks

AWS

Spark/PySpark

Structured Streaming

Kafka

Kinesis

Delta Lake

Medallion Architecture

Unity Catalog

Z-Ordering

Serverless Compute

Lakehouse

Data Governance

CI/CD

Infrastructure as Code

Performance Tuning

Cost Optimization

Lakeflow

Lakebase

LTAP

Zerobus

Tools

Lakeflow

Lakebase

LTAP

Zerobus

Job description

SGI is seeking a Databricks Data Architect to design and lead modern cloud data solutions on Databricks and AWS. This hands-on role focuses on Lakehouse architecture, streaming, governance, and cloud optimization.

You will design enterprise Databricks Lakehouse architectures, batch and real-time data pipelines, and drive performance and cost improvements across workloads. Hybrid onsite/remote collaboration offered.