Sr. Data Engineer

PlanIT Group; United States; On-site; USD 110,000 - 160,000; Full time

Job summary

PlanIT Group designs, develops, optimizes, and maintains data architecture and pipelines using Azure Databricks (PySpark) to enable fast analytics. The team collaborates with data analysts and developers to improve data quality, governance, and security across the organization.

Senior engineers mentor teammates, design end-to-end DataOps workflows, and drive platform scalability across hybrid environments, focusing on healthcare data solutions and compliant data delivery.

Qualifications

• Bachelor's degree in a data-related field or equivalent experience.
• 5+ years in data engineering or architecture, healthcare setting preferred.
• 6 years of related experience for Master’s track, preferred.
• 5 years building cloud data pipelines with Azure Databricks (PySpark) and Azure Data Factory.
• 3 years designing and troubleshooting SQL queries, ETL processes.
• 5 years writing advanced T-SQL; performance tuning and optimization.

Responsibilities

• Design, develop, optimize, and maintain data architecture and pipelines using Azure Databricks (PySpark).
• Provide SME input for data solutions to business use cases and collaborate with analysts and developers.
• Ensure data quality, security, and governance across data products.
• Solicit and ensure compliance with data governance and security requirements for integrated pipelines.
• Lead DataOps and analytical workflow design, architecture, and governing processes.
• Mentor data team members to grow capabilities and reduce delivery risks.

Skills

Azure Databricks

PySpark

SQL

ETL

Data governance

T-SQL

CI/CD

DevOps

Azure Data Factory

C#

PowerShell

Azure Functions

Education

Bachelor's degree in a data-related field

Master's degree Computer Science / Statistics / Applied Mathematics / Information Systems

Tools

Visual Studio

Git

Jenkins

Azure DevOps

Job description

Designs, develops, optimizes, and maintains data architecture and pipelines that adhere to Extraction, Transformation, Loading (ETL) principles and best practices using Azure Databricks (PySpark)Works to identify business problems that can be solved and product features that can be enhanced by data engineering and analytics

Provides subject matter expertise as a product team member, for identification, estimation, and implementation of data solutions to business use cases

Collaborates with data analysts, key data consumers and application developers to optimize models and algorithms for data quality, security, and governance

Solicits, understands, and ensures compliance with data governance and data security requirements while creating and improving integrated and reusable data pipelines, to enable faster data access and improved time-to-solution for data and analytics initiatives

Leads engineering communities of practice (CoP) to expand influence and quality across the digital organization

Designs end to end DataOps and analytical workflows, including overall architecture, capabilities, platforms, tools, and governing processes

Mentors' data team members to grow capabilities and mitigate product delivery risks

Skills and Experience

Bachelors degree and 5 years' experience in a Data Engineering role, in a healthcare setting (required) or 7 years of related experience in lieu of degree. (required)

Master's degree Computer Science, Statistics, Applied mathematics, or information systems and 6 years of related experience or 11 years of related experience and no degree. (preferred)

Minimum Years of Experience

• 5 years' experience in a data engineering or architecture role (required)
• 5 years' experience designing, building, deploying, and troubleshooting data extract and load jobs (required)

5+ years' experience in building cloud data pipelines using Azure Databricks (PySpark) and Azure data factory

• 3 years' experience building DB objects, authoring, and troubleshooting SQL queries, stored procedures and ETL processes (required)
• 5 years' experience writing complex T-SQL queries, strong knowledge of DML, DDL, triggers, common table expressions (CTE), query tuning and performance optimization (required)

3 years' experience foundational data management practices, including:

• Implement Delta Lake architectures, including medallion data models (bronze/silver/gold layers)
• Demonstrated ability to design, build, manage and automate pipelines for data structures encompassing data transformation, data models, schemas, metadata, and workload management
• Build reusable frameworks for metadata-driven ingestion, orchestration, and error handling.

3 years' experience working with DevOps and continuous integration capabilities for version control, build automation, testing, and release management (e.g., Visual Studio, Git, Jenkins, Azure DevOps) (required)

Other Knowledge, Skills and Abilities Required

License/Certification - Azure Data Engineer Associate (preferred)

1 year experience working with C#, PowerShell, Azure functions and logic apps, Azure Key Vault, .Net core, Azure PaaS (preferred)

3 years' experience Exposure to hybrid deployments, cloud and on premise, with demonstrated ability to work across multiple deployment environments, operating systems (preferred)