Azure Databricks Engineer — Lakehouse & Data Pipelines Expert

MACHINE LEARNING TECHNOLOGIES LLC; Dallas (TX); On-site; USD 72,000 - 102,000; Full time

Benefits offered by this job

Job summary

MACHINE LEARNING TECHNOLOGIES LLC in Dallas, TX is seeking an experienced Azure Databricks Engineer to design and optimize scalable data pipelines and Lakehouse architectures. You will work with PySpark, Python, Spark SQL, and Azure Data Services to deliver robust data solutions.

You will collaborate with data architects, analysts, and business stakeholders, implement CI/CD pipelines, and ensure quality, performance, and cost efficiency across large-scale data processing projects.

Qualifications

• Hands-on experience building scalable data pipelines with Azure Databricks.
• Strong Python and PySpark programming skills.
• Experience with Delta Lake and Lakehouse architecture.
• Experience with Azure Data Factory for orchestration.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks.
• Build ETL/ELT workflows using PySpark, Python, and Spark SQL.
• Develop and optimize complex SQL queries and data transformations.
• Work with Azure Data Factory (ADF) for orchestration and workflow automation.
• Implement data Lakehouse architecture using Delta Lake.
• Work with Azure Data Lake Storage Gen2 (ADLS).
• Perform data modelling, data integration, and data migration.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Implement data quality checks, monitoring, and error handling.
• Collaborate with data architects, analysts, and business stakeholders.
• Implement CI/CD pipelines and deployment processes.

Skills

Azure Databricks

Python

PySpark

Spark SQL

Azure Data Factory

ADLS Gen2

Delta Lake

SQL

Azure DevOps/CI-CD

Data pipeline optimization

Tools

Unity Catalog

Databricks Workflows

Databricks Asset Bundles

Azure Synapse Analytics

Terraform

Kafka/Event Hubs

Job description

MACHINE LEARNING TECHNOLOGIES LLC in Dallas, TX is seeking an experienced Azure Databricks Engineer to design and optimize scalable data pipelines and Lakehouse architectures. You will work with PySpark, Python, Spark SQL, and Azure Data Services to deliver robust data solutions.

You will collaborate with data architects, analysts, and business stakeholders, implement CI/CD pipelines, and ensure quality, performance, and cost efficiency across large-scale data processing projects.