Data Engineer: Databricks & Azure Data Factory

Artifintech; Houston (TX); In loco; USD 89.000 - 126.000; Tempo pieno

Descrizione del lavoro

A Data Engineer role in Houston focuses on building and maintaining data pipelines using Databricks and Azure Data Factory (ADF). The role involves ETL/ELT development, data processing, and quality control across data sources and storage layers.

Responsibilities include writing SQL, using PySpark or Python for large-scale processing, and ensuring reliability in data lakes or warehouses within an Azure cloud environment.

Competenze

• Experience with Databricks and Azure Data Factory (ADF).
• Strong understanding of ETL/ELT concepts.
• Good knowledge of SQL and data transformations.
• Basic experience with Python or PySpark.
• Understanding of Azure cloud services and data lake concepts.

Mansioni

• Develop and maintain ETL/ELT pipelines using Azure Data Factory (ADF).
• Work with Databricks for data processing and transformation.
• Write and optimize SQL queries for data extraction and transformation.
• Work with PySpark or Python for large-scale data processing.
• Integrate data from multiple sources into data warehouses or data lakes.
• Monitor, troubleshoot, and optimize data pipelines.
• Collaborate with teams to ensure data quality and reliability.

Conoscenze

Databricks

Azure Data Factory

ETL/ELT

SQL

Python/PySpark

Azure Data Lake

Descrizione del lavoro

A Data Engineer role in Houston focuses on building and maintaining data pipelines using Databricks and Azure Data Factory (ADF). The role involves ETL/ELT development, data processing, and quality control across data sources and storage layers.

Responsibilities include writing SQL, using PySpark or Python for large-scale processing, and ensuring reliability in data lakes or warehouses within an Azure cloud environment.