Senior Azure Databricks Lead - Real-Time Data Pipelines

ZEUS SOLUTIONS INC; Houston (TX); On-site; USD 140,000 - 180,000; Full time

Job summary

ZEUS SOLUTIONS INC in Houston seeks a hands-on Databricks Lead with 8+ years of data engineering experience. You will own real-time and batch data pipelines, optimize Spark workloads, and mentor engineers to build cost-effective, scalable solutions.

The role emphasizes Spark internals, declarative pipelines, and Delta Live Tables, with collaboration across data, analytics, and product teams to deliver robust data platforms.

Qualifications

• 8+ years of hands-on experience in data engineering or a related field.
• Extensive hands-on experience with Azure Databricks and PySpark.
• Strong programming background and knowledge of distributed data processing beyond traditional ETL.
• Proven expertise in Spark performance tuning and optimization.
• Experience with cost management within Databricks.
• Familiarity with Lakehouse Declarative Pipelines (LDP) and Delta Live Tables (DLT).
• Excellent communication and ability to collaborate with cross-functional teams.
• Willingness to work onsite in Houston

Responsibilities

• Lead the design and development of scalable, high-performance data pipelines, streaming tables, and Delta Live Tables (DLT) using Azure Databricks and PySpark.
• Drive Spark performance tuning and cost-optimization strategies within the Databricks environment.
• Build and manage real-time and batch workflows using Structured Streaming.
• Leverage Delta Live Tables (DLT) and Lakehouse Declarative Pipelines (LDP) to build scalable, reliable data pipelines.
• Collaborate with data architects, analysts, and business stakeholders to deliver robust data solutions.
• Provide technical leadership through mentorship, code reviews, and architectural guidance.
• Ensure system reliability and performance through proactive monitoring and engineering best practices.

Skills

Azure Databricks

PySpark

Structured Streaming

Spark tuning

Cost optimization

Delta Live Tables

Lakehouse Pipelines

Distributed data processing

Tools

Azure Data Factory

Azure Synapse

Delta Lake

Job description

ZEUS SOLUTIONS INC in Houston seeks a hands-on Databricks Lead with 8+ years of data engineering experience. You will own real-time and batch data pipelines, optimize Spark workloads, and mentor engineers to build cost-effective, scalable solutions.

The role emphasizes Spark internals, declarative pipelines, and Delta Live Tables, with collaboration across data, analytics, and product teams to deliver robust data platforms.