Data Engineer – Azure Databricks

TechDigital Group; Cincinnati (OH); On-site; USD 80,000 - 110,000; Full time

Job summary

A leading company is seeking a skilled data engineer to design and implement scalable data pipelines on Azure Databricks. The role involves optimizing ETL/ELT workflows and ensuring best practices in data governance and compliance. The ideal candidate will leverage technologies like Apache Spark and Delta Lake to build high-performance data solutions.

Qualifications

• Experience with Azure Databricks and ETL/ELT workflows.
• Proficient in optimizing Azure SQL queries and indexing strategies.

Responsibilities

• Design and implement scalable data pipelines on Azure Databricks.
• Optimize ETL/ELT workflows for efficiency in data processing.
• Lead migration of Azure SQL to Azure Databricks.

Skills

Data Governance

Data Processing

ETL/ELT

Apache Spark

Azure Databricks

Job description

Design and implement scalable data pipelines and architectures on Azure Databricks.
Optimize ETL/ELT workflows, ensuring efficiency in data processing, storage, and retrieval.
Leverage Apache Spark, Delta Lake, and Azure-native services to build high-performance data solutions.
Ensure best practices in data governance, security, and compliance within Azure environments.
Troubleshoot and fine-tune Spark jobs for optimal performance and cost efficiency.
Lead the migration of Azure SQL to Azure Databricks, ensuring a seamless transition of data workloads.
Design and implement scalable data pipelines to extract, transform, and load (ETL/ELT) data from Azure SQL into Databricks Delta Lake.
Optimize Azure SQL queries and indexing strategies before migration to enhance performance in Databricks