Azure DataBricks Engineer

MasterApp Labs; Milton (GA); On-site; USD 140,000 - 190,000; Full time

Job summary

MasterApp Labs in Atlanta, GA is seeking a Senior Azure Databricks Engineer to design, develop, and optimize enterprise-scale data solutions using Azure Databricks, Spark, and Azure services. You will build scalable data pipelines, implement Delta Lake and Lakehouse architectures, and drive governance, security, and performance.

You will collaborate with data scientists, architects, and analysts to deliver real-time and batch processing while ensuring CI/CD deployment and operational excellence

Qualifications

• Bachelor’s degree in CS/IT/Engineering or related field.
• 6+ years of experience in Data Engineering.
• 3+ years hands-on Azure Databricks experience.
• Strong experience with Azure Databricks, Spark, PySpark, Python, SQL, Delta Lake.
• Experience with Azure Data Factory (ADF).
• Experience with ADLS Gen2 and data lakehouse architectures.
• Experience with large-scale datasets and distributed computing.
• Excellent communication and collaboration skills.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks and PySpark.
• Build and support ETL/ELT frameworks for large-scale data.
• Develop batch and real-time data processing solutions.
• Implement Delta Lake and Medallion Architecture (Bronze/Silver/Gold).
• Integrate data solutions with Azure Data Factory, ADLS Gen2, Synapse, Event Hub, and other Azure services.
• Optimize Spark workloads for performance, reliability, and cost.
• Implement data quality checks, monitoring, and alerts.
• Collaborate with stakeholders to deliver data solutions.
• Establish data governance, security, and compliance standards.
• Support CI/CD and automated deployment; troubleshoot production data pipelines.

Skills

Azure Databricks

Apache Spark

PySpark

Python

SQL

Delta Lake

Git

Education

Bachelor's degree in computer science, Information Technology, Engineering, or a related field

Tools

Azure Data Factory

ADLS Gen2

Job description

Azure Databricks Engineer

Location: Atlanta, GA (Onsite)

Employment Type: Full-Time

Experience Required: 6+ Years Overall | 3+ Years Azure Databricks Experience

About The Role

We are seeking a highly skilled Senior Azure Databricks Engineer to join our Data & Analytics team in Atlanta. The ideal candidate will have extensive experience designing, developing, and optimizing enterprise-scale data solutions using Azure Databricks, Apache Spark, and Microsoft Azure services.

This role will be responsible for building scalable data pipelines, implementing Lakehouse architectures, enabling real-time and batch data processing, and driving best practices for performance, governance, and security.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks and PySpark.
• Build and support ETL/ELT frameworks for large-scale structured and unstructured data.
• Develop batch and real-time data processing solutions.
• Implement and maintain Delta Lake and Medallion Architecture (Bronze, Silver, Gold).
• Integrate data solutions with Azure Data Factory, ADLS Gen2, Azure Synapse, Event Hub, and other Azure services.
• Optimize Spark workloads for performance, reliability, and cost efficiency.
• Implement data quality checks, monitoring, and alerting processes.
• Collaborate with business stakeholders, architects, analysts, and data scientists to deliver data solutions.
• Establish and enforce data governance, security, and compliance standards.
• Support CI/CD pipelines and automated deployment processes.Troubleshoot and resolve production issues related to data pipelines and Databricks workloads.
• Document technical solutions, architectural decisions, and operational procedures.

Required Qualifications

• Bachelor’s degree in computer science, Information Technology, Engineering, or a related field.
• 6+ years of experience in Data Engineering.
• 3+ years of hands-on experience with Azure Databricks.
• Strong experience with:
  • Azure Databricks
  • Apache Spark
  • PySpark
  • Python
  • SQL
  • Delta Lake
• Experience with Azure Data Factory (ADF).
• Experience with Azure Data Lake Storage Gen2 (ADLS Gen2).
• Strong understanding of Data Warehousing, Data Lakes, and Lakehouse architectures.
• Experience working with large-scale datasets and distributed computing environments.
• Strong analytical, troubleshooting, and problem-solving skills.
• Excellent communication and collaboration abilities.

Preferred Qualifications

• Experience with Azure Synapse Analytics.
• Experience working with Event Hub or Kafka.
• Knowledge of Unity Catalog and data governance frameworks.
• Experience with Delta Live Tables (DLT).
• Experience with Terraform, ARM Templates, or Infrastructure as Code (IaC).
• Experience with Azure DevOps or GitHub Actions.
• Knowledge of Power BI and modern analytics platforms.
• Cloud certifications such as:
  • Microsoft Azure Data Engineer Associate (DP-203)
  • Databricks Certified Data Engineer Associate/Professional

Required

Technical Skills

• Azure Databricks
• Apache Spark
• PySpark
• Python
• SQL
• Azure Data Factory
• ADLS Gen2
• Delta Lake
• Git

What We're Looking For

• Lakehouse Architecture
• Spark Performance Tuning
• Enterprise Data Engineering
• Data Pipeline Development
• Real-Time Data Processing
• Data Governance and Security
• CDC (Change Data Capture) Implementations
• Production Support and Troubleshooting
• Azure Cloud Data Services
• Cost Optimization and Scalability

Why Join Us?

• Work on enterprise-scale cloud data modernization initiatives.
• Collaborate with experienced data and cloud professionals.
• Exposure to cutting-edge Azure and Databricks technologies.
• Opportunity to drive impactful business solutions through data.
• Long-term growth and career advancement opportunities.