Senior Data Engineer – Azure Databricks & Spark

UST; Alpharetta (GA); On-site; USD 92,000 - 138,000; Full time

Benefits offered by this job

401(k) with employer matching

Medical, dental, and vision insurance

Paid time off and holidays

Job summary

UST in Alpharetta, Georgia is seeking a Senior Data Engineer to design and optimize batch and streaming pipelines on Azure Databricks. You will lead Databricks workflows, implement partitioning strategies, and enable CDC/incremental processing to power scalable data platforms.

The role requires 8+ years of data engineering experience, strong Spark fundamentals, and hands-on knowledge of Databricks, PySpark, and Azure Data Factory.

Qualifications

• 5+ years building batch and/or streaming pipelines with Spark
• Nice to have: Performance tuning: broadcast joins, AQE, skew mitigation, shuffle optimization
• Nice to have: Structured Streaming and Auto Loader for incremental ingestion
• Nice to have: OPTIMIZE, Z ORDER, VACUUM and retention policies
• Nice to have: Compaction and small files mitigation strategies
• Nice to have (optional) Scala, dbt, Kafka/Event Hubs, ADF/Airflow, Terraform, Power BI, data mesh concepts, SCD patterns
• Total experience: 8+ years

Responsibilities

• Partitioning and file layout to match read patterns
• Designing tables for CDC and incremental processing
• Documentation and stakeholder communication
• Databricks Jobs/Workflows: task orchestration, cluster policies, retries, parameters
• Git integration (Repos), CI/CD, and environment promotion

Skills

Databricks

PySpark

Azure Data Factory

Job description

UST in Alpharetta, Georgia is seeking a Senior Data Engineer to design and optimize batch and streaming pipelines on Azure Databricks. You will lead Databricks workflows, implement partitioning strategies, and enable CDC/incremental processing to power scalable data platforms.

The role requires 8+ years of data engineering experience, strong Spark fundamentals, and hands-on knowledge of Databricks, PySpark, and Azure Data Factory.