Senior Data Engineer - Azure Databricks & DataOps Lead

PlanIT Group; United States; On-site; USD 110,000 - 160,000; Full time

Job summary

PlanIT Group designs, develops, optimizes, and maintains data architecture and pipelines using Azure Databricks (PySpark) to enable fast analytics. The team collaborates with data analysts and developers to improve data quality, governance, and security across the organization.

Senior engineers mentor teammates, design end-to-end DataOps workflows, and drive platform scalability across hybrid environments, focusing on healthcare data solutions and compliant data delivery.

Qualifications

• Bachelor's degree in a data-related field or equivalent experience.
• 5+ years in data engineering or architecture, healthcare setting preferred.
• 6 years of related experience for Master’s track, preferred.
• 5 years building cloud data pipelines with Azure Databricks (PySpark) and Azure Data Factory.
• 3 years designing and troubleshooting SQL queries, ETL processes.
• 5 years writing advanced T-SQL; performance tuning and optimization.

Responsibilities

• Design, develop, optimize, and maintain data architecture and pipelines using Azure Databricks (PySpark).
• Provide SME input for data solutions to business use cases and collaborate with analysts and developers.
• Ensure data quality, security, and governance across data products.
• Solicit and ensure compliance with data governance and security requirements for integrated pipelines.
• Lead DataOps and analytical workflow design, architecture, and governing processes.
• Mentor data team members to grow capabilities and reduce delivery risks.

Skills

Azure Databricks

PySpark

SQL

ETL

Data governance

T-SQL

CI/CD

DevOps

Azure Data Factory

C#

PowerShell

Azure Functions

Education

Bachelor's degree in a data-related field

Master's degree Computer Science / Statistics / Applied Mathematics / Information Systems

Tools

Visual Studio

Git

Jenkins

Azure DevOps

Job description

PlanIT Group designs, develops, optimizes, and maintains data architecture and pipelines using Azure Databricks (PySpark) to enable fast analytics. The team collaborates with data analysts and developers to improve data quality, governance, and security across the organization.

Senior engineers mentor teammates, design end-to-end DataOps workflows, and drive platform scalability across hybrid environments, focusing on healthcare data solutions and compliant data delivery.