Data Engineer: Databricks + ADF Pipelines

Machine Learning Technologies, LLC; Atlanta (GA); On-site; USD 110,000 - 150,000; Full time

Job summary

Machine Learning Technologies, LLC is seeking a Data Engineer in Atlanta to develop and maintain scalable data pipelines using Databricks and Azure Data Factory. You will optimize ETL processes, write efficient SQL, and leverage PySpark/Python for large-scale data tasks, ensuring data quality across warehouses and lakes.

You will collaborate with cross-functional teams to deliver reliable data assets and support analytics initiatives in a Microsoft Azure environment.

Qualifications

• Experience with Databricks and Azure Data Factory.
• Strong ETL/ELT concepts understanding.
• Proficient in SQL and data transformations.
• Basic experience with Python or PySpark.
• Familiarity with Azure cloud services and data lake concepts.

Responsibilities

• Develop and maintain ETL/ELT pipelines using Azure Data Factory (ADF).
• Work with Databricks for data processing and transformation.
• Write and optimize SQL queries for data extraction and transformation.
• Work with PySpark or Python for large-scale data processing.
• Integrate data from multiple sources into data warehouses or data lakes.
• Monitor, troubleshoot, and optimize data pipelines.
• Collaborate with teams to ensure data quality and reliability.

Skills

Databricks

Azure Data Factory

SQL

Python

PySpark

Job description

Machine Learning Technologies, LLC is seeking a Data Engineer in Atlanta to develop and maintain scalable data pipelines using Databricks and Azure Data Factory. You will optimize ETL processes, write efficient SQL, and leverage PySpark/Python for large-scale data tasks, ensuring data quality across warehouses and lakes.

You will collaborate with cross-functional teams to deliver reliable data assets and support analytics initiatives in a Microsoft Azure environment.