Azure Data Engineer

Envision

Indianapolis (IN)

On-site

USD 140,000 - 190,000

Full time

16 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Envision seeks an experienced Data Engineering Tech Lead to design, develop, and maintain scalable data lakehouse architectures on Azure, driving batch and near-real-time processing across enterprise sources. You will enforce governance, implement secure access controls, and mentor teams in data transformation within Healthcare.

You will lead architecture, optimize Spark workloads, and build modular workflows with ADF and Databricks. Strong Python and governance experience are essential.

Qualifications

  • 12+ years of data engineering/IT experience.
  • 5+ years on Azure cloud platform.
  • 4+ years with Azure Databricks and PySpark.
  • 3+ years with Azure Data Factory (ADF).
  • 3+ years with ADLS Gen2.
  • 3+ years with Azure SQL.
  • 2+ years with Databricks Workflows and Unity Catalog governance.
  • 2+ years modular Python programming, package builds, testing (PyTest/unittest).

Responsibilities

  • Design, build, and optimize scalable Lakehouse solutions.
  • Ingest data from ERP platforms, APIs, relational databases, NoSQL stores, and on-prem sources.
  • Optimize Spark jobs and large-scale data processing.
  • Implement data access controls, lineage, audit trails, and security with Unity Catalog.
  • Architect Bronze-Silver-Gold storage on ADLS Gen2 with RBAC/ACLs.
  • Build modular orchestration workflows using ADF and Databricks Workflows.
  • Author HLD/LLD, diagrams, and technical specs.
  • Establish coding standards and code reviews; integrate static analysis tools.
  • Build automated deployment pipelines with Azure DevOps, Git, and IaC.
  • Mentor engineers on design, optimization, and best practices.
  • Identity & CI/CD: Azure AD, GitHub Actions.
  • CDC & Migration: Debezium; Synapse to Databricks migration experience.
  • Multi-Cloud Storage: exposure to GCS.

Skills

Azure Databricks
PySpark
Azure Data Factory
ADLS Gen2
Azure SQL
Databricks Workflows
Unity Catalog governance
Python programming
Spark tuning
Data Governance

Tools

Delta Lake
Unity Catalog

Job description

We are seeking an experienced Data Engineering Tech Lead to lead the design, development, optimization, and maintenance of scalable enterprise data engineering and Lakehouse solutions on the Azure cloud platform. In this role, you will drive both batch and near-real-time data processing across complex enterprise data sources, enforce robust data governance, and mentor engineering teams to enable data transformation in the Healthcare sector.

Key Tech Stack & Environment
  • Governance & Architecture: Unity Catalog, Delta Lake, Medallion Architecture (Bronze/Silver/Gold)
Must-Have Technical Qualifications
  • 12+ overall years of experience in Data Engineering / IT industry roles.
  • 5+ years of hands‑on experience across the Azure Cloud platform ecosystem.
  • 4+ years of core experience with Azure Databricks and PySpark.
  • 3+ years of experience with Azure Data Factory (ADF).
  • 3+ years of experience with ADLS Gen2.
  • 3+ years of experience with Azure SQL.
  • 2+ years of experience in Databricks Workflows and Unity Catalog governance.
  • 2+ years of experience in modular Python programming, package builds, and testing (e.g., PyTest/unittest).
Key Responsibilities
1. Data Engineering & Lakehouse Architecture
  • Design, build, and optimize scalable Lakehouse solutions ingesting structured, semi-structured, and unstructured data from ERP platforms, APIs, relational databases, NoSQL stores, and on-premises sources.
  • Optimize Spark jobs and handle large-scale data processing using DataFrame and RDD APIs.
  • Implement performance tuning, partitioning strategies, file format optimizations (Parquet/Delta), and Spark SQL tuning.
  • Implement data access controls, object lineage, audit trails, and security policies using Unity Catalog integrated with Azure AD.
  • Architect multi-layered storage patterns (Bronze-Silver-Gold) on ADLS Gen2 using lifecycle policies, RBAC/ACLs, and folder-level security.
  • Build modular, parameterized orchestration workflows using ADF and Databricks Workflows (custom activities, triggers, dynamic expressions, error handling).
3. Technical Leadership & DevOps
  • Author High-Level Design (HLD) and Low-Level Design (LLD) documents, architectural diagrams, and technical specifications.
  • Establish coding standards, peer code review processes, and integration with static analysis tools (e.g., SonarQube, PyLint).
  • Build automated deployment pipelines using Azure DevOps, Git, and Infrastructure as Code (ARM/Bicep/Terraform).
  • Mentor junior and mid-level engineers on technical design, optimization, and best practices.
Secondary / Good-to-Have Skills
  • Identity & CI/CD: Microsoft Entra ID (Azure AD), GitHub Actions.
  • CDC & Migration: Debezium CDC implementation; Synapse to Databricks migration experience.
  • Multi-Cloud Storage: Exposure to Google Cloud Storage (GCS).
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Azure Data Engineer
Azure Data Engineer

Aroha Technologies, Inc • Greenfield (IN)

On-site
USD 120,000 - 180,000
Senior Data Engineer
Senior Data Engineer

Confidentiel • Pittsburgh

Hybrid
USD 150,000 - 190,000
Cloud Platform Manager
Cloud Platform Manager

Confidentiel • Pittsburgh

Hybrid
USD 140,000 - 190,000
Director Data Engineering
Director Data Engineering

Straive • United States

Remote
USD 180,000 - 240,000
Lead Data Engineer
Lead Data Engineer

Accion Labs • Dallas (TX)

On-site
USD 140,000 - 200,000
Azure Data Engineer
Azure Data Engineer

Recru • Fort Worth (TX), Arlington (TX), Dallas (TX)

On-site
USD 120,000 - 150,000
Azure Data Lakehouse Tech Lead
Azure Data Lakehouse Tech Lead

Envision • Indianapolis (IN)

On-site
USD 140,000 - 190,000
Lead Databricks Data Engineer
Lead Databricks Data Engineer

EXL Talent Acquisition Team • New York (NY)

On-site
USD 180,000 - 240,000
Lead Databricks Data Engineer
Lead Databricks Data Engineer

Fa Ewjt Saasfaprod1 • New York (NY)

On-site
USD 140,000 - 170,000
Lead Azure Data Engineer
Lead Azure Data Engineer

Methodhub • Fremont (CA)

Hybrid
USD 140,000 - 210,000