Data Engineer — Databricks & ADF Pipelines

Quantum Technologies. LLC; Atlanta (GA); On-site; USD 90,000 - 150,000; Full time

Job summary

Quantum Technologies LLC is seeking a Data Engineer to develop, maintain, and optimize data pipelines using Databricks and Azure Data Factory. You will implement ETL/ELT processes, write SQL queries, and leverage PySpark/Python for large-scale data processing.

The role involves integrating data from multiple sources into data lakes or warehouses and ensuring data reliability. Ideal candidates will have hands-on experience with Databricks, ADF, SQL, and Azure cloud services, with knowledge of

Qualifications

• Experience with Databricks and Azure Data Factory (ADF).
• Strong understanding of ETL/ELT concepts.
• Good knowledge of SQL and data transformations.
• Basic experience with Python or PySpark.
• Understanding of Azure cloud services and data lake concepts.

Responsibilities

• Develop and maintain ETL/ELT pipelines using Azure Data Factory (ADF).
• Work with Databricks for data processing and transformation.
• Write and optimize SQL queries for data extraction and transformation.
• Work with PySpark or Python for large-scale data processing.
• Integrate data from multiple sources into data warehouses or data lakes.
• Monitor, troubleshoot, and optimize data pipelines.
• Collaborate with teams to ensure data quality and reliability.

Skills

ETL/ELT concepts

SQL

PySpark

Azure cloud concepts

Data lake concepts

Tools

Databricks

Azure Data Factory (ADF)

Job description

Quantum Technologies LLC is seeking a Data Engineer to develop, maintain, and optimize data pipelines using Databricks and Azure Data Factory. You will implement ETL/ELT processes, write SQL queries, and leverage PySpark/Python for large-scale data processing.

The role involves integrating data from multiple sources into data lakes or warehouses and ensuring data reliability. Ideal candidates will have hands-on experience with Databricks, ADF, SQL, and Azure cloud services, with knowledge of