Azure Data Engineer: Cloud Data Pipelines & Spark Pro

Valiant Technologies LLC; Austin (TX); In loco; USD 110.000 - 160.000; Tempo pieno

Descrizione del lavoro

Valiant Technologies LLC is seeking a data engineer to design, implement, and optimize cloud-based ETL/ELT pipelines using Azure Data Factory, Databricks, and PySpark. The role emphasizes secure data governance, scalable data lakes, and close collaboration with analytics teams.

Candidates should be proficient in Python, SQL, Spark optimization, and CI/CD deployment with Azure DevOps. This position requires US work authorization and a focus on reliability and data quality.

Competenze

• Experience with cloud-based data platforms and ETL/ELT pipelines.
• Strong understanding of data governance and security practices.

Mansioni

• Design, develop, and maintain scalable ETL/ELT pipelines using Azure Data Factory.
• Build and optimize data processing solutions using Azure Databricks and PySpark.
• Develop data transformation workflows using Python and SQL.
• Implement data ingestion from multiple sources, including databases, APIs, files, and cloud platforms.
• Build and maintain data lake and data warehouse solutions using ADLS Gen2 and Azure Synapse.
• Optimize Spark jobs, SQL queries, and data processing performance.
• Implement data validation, data quality checks, and error-handling mechanisms.
• Work with Delta Lake for reliable and scalable data storage.
• Implement CI/CD pipelines and deployment automation using Azure DevOps.
• Ensure data security, governance, and compliance standards.
• Collaborate with data architects, analysts, and business stakeholders.
• Troubleshoot production data pipeline issues and ensure operational stability.

Conoscenze

SQL

Data integration

Data modeling

ETL/ELT principles

Troubleshooting

Collaboration with stakeholders

Strumenti

Azure Data Factory

Azure Databricks

Python

PySpark

Azure Synapse Analytics

Azure Data Lake Storage Gen2

Delta Lake

Azure DevOps

Git

Descrizione del lavoro

Valiant Technologies LLC is seeking a data engineer to design, implement, and optimize cloud-based ETL/ELT pipelines using Azure Data Factory, Databricks, and PySpark. The role emphasizes secure data governance, scalable data lakes, and close collaboration with analytics teams.

Candidates should be proficient in Python, SQL, Spark optimization, and CI/CD deployment with Azure DevOps. This position requires US work authorization and a focus on reliability and data quality.