Azure Databricks Engineer - Scalable Data Pipelines & Lakehouse

Valiant Technologies LLC; Dallas (TX); Presencial; USD 120 000 - 170 000; Tempo integral

Resumo da oferta

Valiant Technologies LLC is seeking an experienced Azure Databricks Engineer to design, develop, and optimize large-scale data pipelines and Lakehouse architectures on Azure.

You will work with PySpark, Python, Spark SQL, and ADLS Gen2, building scalable data solutions, implementing data quality checks, and enabling CI/CD pipelines. Strong collaboration with data teams and stakeholders is required.

Qualificações

• Hands-on experience with Azure Databricks and Delta Lake architectures.
• Proficient in PySpark, Python, and Spark SQL.
• Experience building scalable data pipelines and Lakehouse solutions on Azure.
• Familiarity with Azure Data Factory for orchestration and ADLS Gen2.
• Knowledge of CI/CD processes and Azure DevOps.

Responsabilidades

• Design, develop, and maintain scalable data pipelines using Azure Databricks.
• Build ETL/ELT workflows with PySpark and Spark SQL.
• Develop and optimize complex SQL queries and data transformations.
• Utilize ADF for orchestration and workflow automation.
• Work with ADLS Gen2 for storage and data lake governance.
• Model data, integrate sources, and plan migrations.
• Optimize Spark jobs for performance and cost.
• Implement data quality checks and monitoring.
• Collaborate with architects, analysts, and stakeholders.
• Implement CI/CD pipelines and deployment processes.

Conhecimentos

Azure Databricks

PySpark

Python

Spark SQL

SQL

Delta Lake

Azure Data Factory

Azure Data Lake Storage Gen2

Azure DevOps

CI/CD

Ferramentas

Databricks Workflows

Unity Catalog

Terraform

Kafka/Event Hubs

Descrição da oferta de emprego

Valiant Technologies LLC is seeking an experienced Azure Databricks Engineer to design, develop, and optimize large-scale data pipelines and Lakehouse architectures on Azure.

You will work with PySpark, Python, Spark SQL, and ADLS Gen2, building scalable data solutions, implementing data quality checks, and enabling CI/CD pipelines. Strong collaboration with data teams and stakeholders is required.