Data Engineer: Databricks & ADF Pipelines

Valiant Technologies LLC; Atlanta (GA); In loco; USD 90.000 - 130.000; Tempo pieno

Descrizione del lavoro

Valiant Technologies LLC is seeking a Data Engineer in Atlanta, GA to design, build, and optimize data pipelines using Databricks and Azure Data Factory (ADF). The role requires SQL proficiency and Python/PySpark for large-scale processing, with a focus on data quality and reliability.

Responsibilities include developing ETL/ELT pipelines, transforming data, and collaborating with teams to ensure robust data integration across data warehouses and data lakes.

Competenze

• Understanding ETL/ELT concepts and best practices.
• Strong SQL knowledge and data transformation skills.
• Basic to intermediate Python or PySpark for data processing.
• Familiarity with Azure cloud services and data lake concepts.

Mansioni

• Develop and maintain ETL/ELT pipelines using Azure Data Factory (ADF).
• Work with Databricks for data processing and transformation.
• Write and optimize SQL queries for data extraction and transformation.
• Use PySpark or Python for large-scale data processing.
• Integrate data from multiple sources into data warehouses or data lakes.
• Monitor, troubleshoot, and optimize data pipelines.
• Collaborate with teams to ensure data quality and reliability.

Conoscenze

ETL/ELT concepts

SQL

Python/PySpark

Strumenti

Databricks

Azure Data Factory (ADF)

Descrizione del lavoro

Valiant Technologies LLC is seeking a Data Engineer in Atlanta, GA to design, build, and optimize data pipelines using Databricks and Azure Data Factory (ADF). The role requires SQL proficiency and Python/PySpark for large-scale processing, with a focus on data quality and reliability.

Responsibilities include developing ETL/ELT pipelines, transforming data, and collaborating with teams to ensure robust data integration across data warehouses and data lakes.