Data Engineer: Databricks & Azure Data Factory

MACHINE LEARNING TECHNOLOGIES LLC; Houston (TX); On-site; USD 89,000 - 126,000; Full time

Job summary

MACHINE LEARNING TECHNOLOGIES LLC in Houston, TX is seeking a Data Engineer to design, develop, and optimize data pipelines using Databricks and Azure Data Factory. The role emphasizes ETL/ELT processes, data quality, and scalable transformation.

You will write SQL, use PySpark/Python for large data tasks, and collaborate with teams to integrate diverse data sources into data lakes or warehouses, ensuring reliability and performance.

Qualifications

• Develop and maintain ETL/ELT pipelines using Azure Data Factory (ADF).
• Work with Databricks for data processing and transformation.
• Write and optimize SQL queries for data extraction and transformation.
• Work with PySpark or Python for large-scale data processing.
• Integrate data from multiple sources into data warehouses or data lakes.
• Monitor, troubleshoot, and optimize data pipelines.
• Collaborate with teams to ensure data quality and reliability.

Responsibilities

• Develop and maintain ETL/ELT pipelines using Azure Data Factory (ADF).
• Work with Databricks for data processing and transformation.
• Write and optimize SQL queries for data extraction and transformation.
• Work with PySpark or Python for large-scale data processing.
• Integrate data from multiple sources into data warehouses or data lakes.
• Monitor, troubleshoot, and optimize data pipelines.
• Collaborate with teams to ensure data quality and reliability.

Skills

Databricks

Azure Data Factory

SQL

Python

PySpark

ETL/ELT

Tools

Databricks

Azure Data Factory

Job description

MACHINE LEARNING TECHNOLOGIES LLC in Houston, TX is seeking a Data Engineer to design, develop, and optimize data pipelines using Databricks and Azure Data Factory. The role emphasizes ETL/ELT processes, data quality, and scalable transformation.

You will write SQL, use PySpark/Python for large data tasks, and collaborate with teams to integrate diverse data sources into data lakes or warehouses, ensuring reliability and performance.