Senior Azure Databricks Engineer Lakehouse & Real-Time Data

MasterApp Labs; Milton (GA); On-site; USD 140,000 - 190,000; Full time

Job summary

MasterApp Labs in Atlanta, GA is seeking a Senior Azure Databricks Engineer to design, develop, and optimize enterprise-scale data solutions using Azure Databricks, Spark, and Azure services. You will build scalable data pipelines, implement Delta Lake and Lakehouse architectures, and drive governance, security, and performance.

You will collaborate with data scientists, architects, and analysts to deliver real-time and batch processing while ensuring CI/CD deployment and operational excellence

Qualifications

• Bachelor’s degree in CS/IT/Engineering or related field.
• 6+ years of experience in Data Engineering.
• 3+ years hands-on Azure Databricks experience.
• Strong experience with Azure Databricks, Spark, PySpark, Python, SQL, Delta Lake.
• Experience with Azure Data Factory (ADF).
• Experience with ADLS Gen2 and data lakehouse architectures.
• Experience with large-scale datasets and distributed computing.
• Excellent communication and collaboration skills.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks and PySpark.
• Build and support ETL/ELT frameworks for large-scale data.
• Develop batch and real-time data processing solutions.
• Implement Delta Lake and Medallion Architecture (Bronze/Silver/Gold).
• Integrate data solutions with Azure Data Factory, ADLS Gen2, Synapse, Event Hub, and other Azure services.
• Optimize Spark workloads for performance, reliability, and cost.
• Implement data quality checks, monitoring, and alerts.
• Collaborate with stakeholders to deliver data solutions.
• Establish data governance, security, and compliance standards.
• Support CI/CD and automated deployment; troubleshoot production data pipelines.

Skills

Azure Databricks

Apache Spark

PySpark

Python

SQL

Delta Lake

Git

Education

Bachelor's degree in computer science, Information Technology, Engineering, or a related field

Tools

Azure Data Factory

ADLS Gen2

Job description

MasterApp Labs in Atlanta, GA is seeking a Senior Azure Databricks Engineer to design, develop, and optimize enterprise-scale data solutions using Azure Databricks, Spark, and Azure services. You will build scalable data pipelines, implement Delta Lake and Lakehouse architectures, and drive governance, security, and performance.

You will collaborate with data scientists, architects, and analysts to deliver real-time and batch processing while ensuring CI/CD deployment and operational excellence