Azure Data Engineer: Spark, Databricks & ETL Pipelines

Machine Learning Technologies, LLC; Austin (TX); On-site; USD 120,000 - 150,000; Full time

Job summary

Machine Learning Technologies, LLC is seeking a highly skilled data engineer focused on Azure data services in Austin. You will design and implement scalable ETL/ELT pipelines, work with Delta Lake, and optimize Spark jobs and SQL queries to deliver reliable analytics pipelines.

The role collaborates with data architects and business stakeholders to ensure data quality and security. Key responsibilities include building data lakes and data warehouses using ADLS Gen2 and Synapse, implementing

Qualifications

• Strong experience in Microsoft Azure cloud data services.
• Hands-on experience with Azure Data Factory (ADF).
• Strong experience with Azure Databricks.
• Good programming experience in Python and PySpark.
• Strong SQL skills and query optimization.
• Experience with Azure Synapse Analytics.
• Knowledge of Azure Data Lake Storage Gen2 (ADLS).
• Experience with data warehousing and data modeling concepts.
• Hands-on experience with data integration and transformation.
• Experience with Azure DevOps, Git, and CI/CD pipelines.
• Knowledge of data security, governance, and data quality.
• Design, develop, and maintain scalable ETL/ELT pipelines using Azure Data Factory.
• Build and optimize data processing solutions using Azure Databricks and PySpark.
• Develop data transformation workflows using Python and SQL.
• Implement data ingestion from multiple sources, including databases, APIs, files, and cloud platforms.
• Build and maintain data lake and data warehouse solutions using ADLS Gen2 and Azure Synapse.
• Optimize Spark jobs, SQL queries, and data processing performance.
• Implement data validation, data quality checks, and error-handling mechanisms.
• Work with Delta Lake for reliable and scalable data storage.
• Implement CI/CD pipelines and deployment automation using Azure DevOps.
• Ensure data security, governance, and compliance standards.
• Collaborate with data architects, analysts, and business stakeholders.
• Troubleshoot production data pipeline issues and ensure operational stability.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Azure Data Factory.
• Build and optimize data processing solutions using Azure Databricks and PySpark.
• Develop data transformation workflows using Python and SQL.
• Collaborate with data architects and stakeholders to ensure data quality and compliance.
• Troubleshoot production data pipelines to ensure operational stability.

Skills

Azure cloud data services

Python & PySpark

SQL optimization

Data modeling

Data governance

ETL/ELT design

Tools

Azure Data Factory

Azure Databricks

Azure Synapse Analytics

ADLS Gen2

Azure DevOps

Git

Job description

Machine Learning Technologies, LLC is seeking a highly skilled data engineer focused on Azure data services in Austin. You will design and implement scalable ETL/ELT pipelines, work with Delta Lake, and optimize Spark jobs and SQL queries to deliver reliable analytics pipelines.

The role collaborates with data architects and business stakeholders to ensure data quality and security. Key responsibilities include building data lakes and data warehouses using ADLS Gen2 and Synapse, implementing