Azure Data Engineer: Spark, Delta Lake & Pipelines

TechDigital Group; Pleasanton (CA); On-site; USD 120,000 - 170,000; Full time

Job summary

TechDigital Group in Pleasanton, CA seeks an Azure Data Engineer to design scalable data pipelines on Azure Databricks, ADF, and SQL, with Spark PySpark and Delta Lake expertise.

You will build ETL/ELT workflows, optimize queries, and collaborate with cross-functional teams in an Agile environment, contributing to data quality and CI/CD for data workloads.

Qualifications

• Hands-on experience with Azure Databricks and ADF required.
• Strong SQL skills with performance tuning and data transformation.
• Familiarity with Delta Lake, ADLS Gen2 and data warehousing concepts.

Responsibilities

• Design, develop, and maintain data pipelines using Azure Databricks and Delta Lake.
• Build and orchestrate ETL/ELT workflows in Azure Data Factory with pipelines and triggers.
• Write and optimize complex SQL for data transformation and reporting.
• Collaborate with cross-functional teams to deliver reliable data solutions.
• Implement data quality checks, monitoring, and CI/CD for data workloads.

Skills

Spark PySpark

Spark SQL

Delta Lake

Data Modeling

CI/CD

Git/Azure Repos

Azure DevOps

Tools

Azure Databricks (ADB)

Azure Data Factory (ADF)

Azure Analysis Services (AAS)

SQL

Job description

Mandatory Skills: Azure Databricks (ADB), Azure Data Factory (ADF), Azure Analysis Service (AAS), SQL

Job Summary
We are seeking a skilled Azure Data Engineer with strong hands-on expertise in Azure Databricks (ADB), Azure Data Factory (ADF), and SQL. The ideal candidate will design and build scalable data pipelines and transformations on the Azure platform, with solid experience in Spark (PySpark), the Databricks Lakehouse, and ETL/ELT orchestration. Exposure to Delta Lake, data modeling, and CI/CD for data workloads is a plus.

Key Responsibilities
• Design, develop, and maintain data pipelines using Azure Databricks (PySpark/Spark SQL) and Delta Lake.
• Build and orchestrate ETL/ELT workflows in Azure Data Factory, including linked services, datasets, and triggers.
• Write and optimize complex SQL for data transformation, aggregation, and performance tuning.
• Collaborate with cross-functional teams to deliver reliable, high-quality data solutions.
• Implement data quality checks, monitoring, and follow engineering best practices.
• Work in an Agile environment and contribute to sprint planning and retrospectives.

Skill Requirements
• Databricks: Strong proficiency in Azure Databricks, PySpark, Spark SQL, and notebook development.
• Data Integration: Hands-on experience with Azure Data Factory (pipelines, mapping data flows, triggers).
• Database: Expert-level SQL (mandatory) with T-SQL/Spark SQL; query optimization and stored procedures.
• Storage & Format: Experience with Delta Lake, Parquet, and Azure Data Lake Storage (ADLS Gen2).
• Data Modeling: Understanding of dimensional modeling and data warehousing concepts.
• Version Control: Git/Azure Repos or similar tools.
• CI/CD: Familiarity with Azure DevOps pipelines for data deployments.
• Cloud: Working knowledge of the broader Azure ecosystem (Key Vault, Synapse is a plus).