Azure Data Engineer

Aroha Technologies, Inc; Greenfield (IN); On-site; USD 120,000 - 180,000; Full time

Job summary

Aroha Technologies, Inc. in Greenfield, IN is seeking an experienced Data Engineering Tech Lead to design, build, optimize, and maintain scalable Lakehouse solutions on Azure. You will drive batch and near-real-time data processing using Databricks, PySpark, ADF, ADLS Gen2, and Azure SQL.

Lead data pipelines across ERP, API, and relational sources, enforce data governance, and mentor engineers while advancing DevOps practices in the Azure cloud environment.

Qualifications

• 4+ years of experience in Azure Databricks with PySpark.
• 2+ years of experience in Databricks workflow and Unity Catalog.
• 3+ years of experience in ADF (Azure Data Factory).
• 3+ years of experience in ADLS Gen2.
• 3+ years of experience in Azure SQL.
• 5+ years of experience in Azure Cloud platform.
• 2+ years of experience in Python programming and package builds.

Responsibilities

• Lead design, development, and optimization of scalable data pipelines across enterprise sources.
• Ensure high-performance data processing using Databricks, PySpark, and ADF.
• Establish strong data governance, security, and quality across the Lakehouse.
• Provide continuous technical leadership, optimization, and modernization of the data platform.

Skills

Azure Databricks

PySpark

Azure Data Factory

ADLS Gen2

Azure SQL

Azure Cloud

Python

Unity Catalog

Tools

GitHub Actions

Azure DevOps

Job description

Location: Greenfield, IN (Onsite)

Work Arrangement: On-site

Role Summary

We are looking for an experienced Data Engineering Tech Lead with strong expertise in Azure Databricks, PySpark, Azure Data Factory (ADF), ADLS Gen2, Azure SQL, Python, and Azure Cloud. The role will lead the design, development, optimization, and maintenance of scalable enterprise data engineering and Lakehouse solutions, including batch and near-real-time data processing.

Primary (Must-Have Skills)

• 4+ years of experience in Azure Databricks with PySpark.
• 2+ years of experience in Databricks workflow and Unity Catalog.
• 3+ years of experience in ADF (Azure Data Factory).
• 3+ years of experience in ADLS Gen2.
• 3+ years of experience in Azure SQL.
• 5+ years of experience in Azure Cloud platform.
• 2+ years of experience in Python programming and package builds.

This Role Ensures

• Reliable and scalable data pipelines across enterprise data sources.
• High-performance data processing using Databricks, PySpark, and ADF.
• Strong data governance, security, and quality across the Lakehouse.
• Continuous technical leadership, optimization, and modernization of the data platform.

Job Description Of Role Key Technical Skills

• Data management experience handling analytics workloads covering the design, development, and maintenance of Lakehouse solutions sourcing data from platforms such as ERP sources, API sources, relational stores, NoSQL, and on-premises sources using Databricks/PySpark as distributed/big data management services, supporting batch and near-real-time ingestion, transformation, and processing.
• Ability to optimize Spark jobs and manage large-scale data processing using RDD/DataFrame APIs. Demonstrated expertise in partitioning strategies, file format optimization (Parquet/Delta), and Spark SQL tuning. Familiarity with Databricks runtime versions, cluster policies, libraries, and workspace management.
• Skilled in governing and managing data access for Azure Data Lakehouse with Unity Catalog. Experience in configuring data permissions, object lineage, and access policies with Unity Catalog. Understanding of integrating Unity Catalog with Azure AD, external metastores, and audit trails.
• Experience in building efficient orchestration solutions using Azure Data Factory and Databricks Workflows. Ability to design modular, reusable workflows using tasks, triggers, and dependencies. Skilled in using dynamic expressions, parameterized pipelines, custom activities, and triggers.
• Familiarity with integration runtime configurations, pipeline performance tuning, and error-handling strategies.
• Strong experience in implementing secure, hierarchical namespace-based data lake storage for structured/semi-structured data, aligned to bronze-silver-gold layers with ADLS Gen2. Hands‑on experience with lifecycle policies, access control (RBAC/ACLs), and folder-level security. Understanding of best practices in file partitioning, retention management, and storage performance optimization.
• Capable of developing T-SQL queries, stored procedures, and managing metadata layers on Azure SQL.
• Comprehensive experience working across the Azure ecosystem, including networking, security, monitoring, and cost management relevant to data engineering workloads. Understanding of VNets, Private Endpoints, Key Vaults, Managed Identities, and Azure Monitor. Exposure to DevOps tools for deployment automation (e.g., Azure DevOps, ARM/Bicep/Terraform).
• Experience in writing modular, testable Python code used in data transformations, utility functions, and packaging reusable components. Familiarity with Python environments, dependency management (pip/Poetry/Conda), and packaging libraries. Ability to write unit tests using PyTest/unittest and integrate with CI/CD pipelines.
• Lead solution design discussions, mentor junior engineers, and ensure adherence to coding guidelines, design patterns, and peer review processes. Able to prepare design documents for development and guide the team technically. Experience preparing technical design documents, HLDs/LLDs, and architecture diagrams. Familiarity with code quality tools (e.g., SonarQube, pylint) and version control workflows (Git).
• Demonstrates strong verbal and written communication, proactive stakeholder engagement, and a collaborative attitude in cross-functional teams. Ability to articulate technical concepts clearly to both technical and business audiences. Experience working with product owners, QA, and business analysts to translate requirements into deliverables.

Task / Work Updates

• Prior experience working on Agile/Scrum projects with exposure to tools like Jira/Azure DevOps.
• Provides regular updates and is proactive and diligent in carrying out responsibilities.

Expected Outcome

We are seeking a highly skilled Data Engineering specialist with the above-mentioned primary skills to join our dynamic team, which is at the forefront of enabling enterprises in the healthcare sector.

The ideal candidate should be passionate about Data Engineering on Azure Cloud, with a strong focus on DevOps practices in building products for customers.

Effectively communicate and collaborate with internal teams and customers to build code, leverage or create low-level design documents, and align with standard coding principles and guidelines.

Secondary Skills (Good to Have)

• Azure Entra/AD skills and GitHub Actions.
• Orchestration experience using Airflow, Dagster, or Logic Apps.
• Experience working on event-driven architectures using Kafka or Azure Event Hubs.
• Exposure to Google Cloud Pub/Sub.
• Experience developing and maintaining Change Data Capture (CDC) solutions, preferably using Debezium.
• Hands‑on experience with data migration projects, specifically involving Azure Synapse and Databricks Lakehouse.
• Experience managing cloud storage solutions on Azure Data Lake Storage. Experience with Google Cloud Storage will be an advantage.

Primary Skills: Azure Databricks, PySpark, Azure Data Factory (ADF), ADLS Gen2, Azure SQL, Azure Cloud, Python, Unity Catalog.

Secondary Skills: Azure Entra/AD, GitHub Actions, Airflow, Dagster, Logic Apps, Kafka, Azure Event Hubs, Google Cloud Pub/Sub, Debezium, Azure Synapse, Google Cloud Storage.