Senior Data Engineer - Azure Databricks & ADF Pipelines

Holistic Partners, Inc; United States; On-site; USD 120,000 - 160,000; Full time

Job summary

Holistic Partners, Inc. is seeking a Data Engineer to design, develop, and support enterprise Data Lake solutions that drive critical reporting, analytics, and BI initiatives.

You will build scalable data pipelines and optimize processing across the data platform. Responsibilities include implementing pipelines with Azure Data Factory and Azure Databricks, building PySpark and Python ETL, and delivering production-ready CI/CD workflows with Azure DevOps and Git.

Qualifications

• 5+ years of Data Engineering, ETL/ELT, or application development experience.
• Strong hands-on experience with Azure Databricks, Azure Data Factory (ADF), PySpark, Python, and SQL.
• Experience building Databricks notebooks and workspace datasets.
• Azure DevOps, Git, and CI/CD pipeline implementation.
• Experience with enterprise Data Lake and Data Warehouse environments.
• Strong troubleshooting, performance tuning, and production support experience.
• Experience supporting Tableau reporting and analytics solutions.
• Familiarity with Snowflake is highly preferred.

Responsibilities

• Design, develop, and maintain data pipelines using Azure Data Factory and Azure Databricks.
• Build and optimize PySpark and Python solutions for data ingestion, transformation, and enrichment.
• Develop Databricks notebooks and workspace datasets.
• Support Data Lake refresh processes and troubleshoot data quality, performance, and operational issues.
• Develop and maintain CI/CD pipelines using Azure DevOps and Git.
• Collaborate with architects, analysts, engineers, and reporting teams to deliver scalable data solutions.
• Provide production support, monitoring, root cause analysis, and continuous improvement across the data platform.
• Build pipelines and tables that enable the Analytics team to develop dashboards and reports.

Skills

Azure Databricks

Azure Data Factory

PySpark

Python

SQL

CI/CD pipelines

Azure DevOps

Git

Data Lake

Data Warehouse

Tableau

Snowflake

Production support

Tools

Databricks notebooks

Workspace datasets

Job description

Holistic Partners, Inc. is seeking a Data Engineer to design, develop, and support enterprise Data Lake solutions that drive critical reporting, analytics, and BI initiatives.

You will build scalable data pipelines and optimize processing across the data platform. Responsibilities include implementing pipelines with Azure Data Factory and Azure Databricks, building PySpark and Python ETL, and delivering production-ready CI/CD workflows with Azure DevOps and Git.