Azure Databricks Data Engineer — Lakehouse & Spark Expert

Quantum Technologies. LLC; Dallas (TX); On-site; USD 120,000 - 180,000; Full time

Job summary

Quantum Technologies LLC is seeking an experienced Azure Databricks Engineer to design and optimize large-scale data pipelines and Lakehouse architectures in a Dallas area environment. You will leverage Azure Databricks, PySpark, and Spark SQL to deliver scalable solutions and partner with data peers and stakeholders.

The role requires hands-on experience with Delta Lake, ADLS Gen2, and CI/CD deployment practices, with emphasis on performance tuning and data quality.

Qualifications

• 5+ years of experience in data engineering with Azure Databricks.
• Strong Python and PySpark programming skills.
• Experience with Azure Data Factory and ADLS Gen2.
• Knowledge of Delta Lake and Lakehouse architecture.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks.
• Build ETL/ELT workflows using PySpark, Python, and Spark SQL.
• Develop and optimize complex SQL queries and data transformations.
• Work with Azure Data Factory (ADF) for orchestration and workflow automation.
• Work with Azure Data Lake Storage Gen2 (ADLS).
• Perform data modelling, data integration, and data migration.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Implement data quality checks, monitoring, and error handling.
• Collaborate with data architects, analysts, and business stakeholders.
• Implement CI/CD pipelines and deployment processes.

Skills

Azure Databricks

PySpark

Python

Spark SQL

Delta Lake

Azure Data Factory

Azure Data Lake Storage Gen2

SQL

Azure DevOps

CI/CD

Performance tuning

Education

Bachelor's degree in Computer Science or related field

Tools

Terraform

Databricks Workflows

Kafka

Azure Synapse Analytics

Job description

Quantum Technologies LLC is seeking an experienced Azure Databricks Engineer to design and optimize large-scale data pipelines and Lakehouse architectures in a Dallas area environment. You will leverage Azure Databricks, PySpark, and Spark SQL to deliver scalable solutions and partner with data peers and stakeholders.

The role requires hands-on experience with Delta Lake, ADLS Gen2, and CI/CD deployment practices, with emphasis on performance tuning and data quality.