Azure Lakehouse Data Tech Lead

Purple Hires Inc; Indiana; On-site; USD 120,000 - 170,000; Full time

Job summary

Purple Hires Inc is seeking an experienced Azure Data Engineer to lead the design, development, and maintenance of Lakehouse solutions on Azure. The role emphasizes PySpark, Databricks, and robust data governance in a full-time onsite environment in Indiana.

Responsibilities include building scalable data pipelines, optimizing Spark jobs, implementing Unity Catalog governance, and integrating with Azure Data Factory and Databricks Workflows.

Qualifications

• 4+ years of experience in Azure Databricks with PySpark.
• 2+ years of experience in Databricks Workflows and Unity Catalog.
• 3+ years of experience in Azure Data Factory (ADF).
• 3+ years of experience in ADLS Gen2.
• 3+ years of experience in Azure SQL.
• 5+ years of experience in the Azure Cloud platform.
• 2+ years of experience in Python programming and package builds.

Responsibilities

• Handle analytics workloads covering the design, development, and maintenance of Lakehouse solutions.
• Source data from platforms such as ERP systems, APIs, relational databases, NoSQL databases, and on-premises sources using Databricks and PySpark.
• Support batch and near-real-time data ingestion, transformation, and processing.
• Implement scalable and efficient data processing solutions for enterprise data platforms.
• Optimize Spark jobs and manage large-scale data processing using RDD and DataFrame APIs.
• Apply partitioning strategies, file format optimization using Parquet and Delta, and Spark SQL tuning.
• Work with Databricks runtime versions, cluster policies, libraries, and workspace management.
• Govern and manage data access for the Azure Data Lakehouse using Unity Catalog.
• Configure data permissions, object lineage, and access policies.
• Understand integration with Azure AD, external metastores, and audit trails.
• Build efficient orchestration solutions using Azure Data Factory and Databricks Workflows.
• Design modular, reusable workflows using tasks, triggers, and dependencies.
• Implement dynamic expressions, parameterized pipelines, custom activities, and triggers.
• Configure integration runtimes and optimize pipeline performance and error-handling strategies.
• Implement secure, hierarchical namespace-based data lake storage for structured and semi-structured data.
• Develop Bronze, Silver, and Gold data layers using ADLS Gen2.
• Manage lifecycle policies, RBAC, ACLs, and folder-level security.
• Apply best practices for file partitioning, retention management, and storage performance optimization.
• Develop T-SQL queries and stored procedures and manage metadata layers on Azure SQL.
• Write modular, testable Python code for data transformations, utility functions, and reusable components.
• Manage Python environments, dependencies, and package libraries using tools such as pip, Poetry, or Conda.
• Write unit tests using PyTest or unittest and integrate testing into CI/CD pipelines.
• Work across the Azure ecosystem, including networking, security, monitoring, and cost management for data engineering workloads.
• Understand VNets, Private Endpoints, Key Vaults, Managed Identities, and Azure Monitor.
• Use DevOps tools for deployment automation, including Azure DevOps, ARM, Bicep, and Terraform.
• Follow version control workflows using Git and apply code quality tools such as SonarQube and pylint.

Skills

Azure Databricks

PySpark

Azure Cloud

Python

Azure SQL

Tools

Databricks Workflows

Unity Catalog

Azure Data Factory

ADLS Gen2

Azure DevOps

Job description

Purple Hires Inc is seeking an experienced Azure Data Engineer to lead the design, development, and maintenance of Lakehouse solutions on Azure. The role emphasizes PySpark, Databricks, and robust data governance in a full-time onsite environment in Indiana.

Responsibilities include building scalable data pipelines, optimizing Spark jobs, implementing Unity Catalog governance, and integrating with Azure Data Factory and Databricks Workflows.