Lead Data Engineer

Delta System & Software, Inc.; Charlotte (NC); On-site; USD 120,000 - 160,000; Full time

Job summary

Delta System & Software, Inc. in Charlotte, NC is seeking a Lead Data Engineer to architect and optimize scalable data pipelines using Azure Databricks, PySpark, SQL, and Delta Lake. You will drive ETL/ELT workflows, improve CI/CD, and mentor a team of engineers.

The role emphasizes data quality, governance, and secure data management with Unity Catalog and Azure RBAC. Strong Python and Terraform skills are required for automation and deployment.

Qualifications

• Bachelor’s degree in computer science or information systems science, or equivalent experience.
• Three (3) years experience with Azure Databricks platform.
• Five (5) years of comprehensive data warehouse analysis and design experience, with full knowledge of data warehouse methodologies, data modeling and familiarity with operational data stores.
• Five (5) years of experience with ETL/ELT Development, Python, PySpark.
• Python, SQL, and Terraform experience.

Responsibilities

• Lead design, development, and optimization of scalable data pipelines and ETL/ELT workflows using Azure Databricks, PySpark, SQL, and Delta Lake.
• Enhancement of CI/CD pipelines and Azure DevOps, Git, and automated deployment frameworks to streamline development and release processes.
• Ensure data quality and consistency through data integration, transformation, and cleansing.
• Familiarity with security and governance best practices using Unity Catalog, Azure RBAC, managed identities, and data access controls to ensure compliance and secure data management is preferred.
• Responsibility includes leading a team of data engineers and mentoring in best practices.
• Utilize Python for data manipulation, automation, and custom data solutions.
• Collaborate with stakeholders to understand data needs and translate them into technical solutions.
• Participate in on-call rotation and assist with after-hours issues, as required.
• Perform other duties as assigned

Skills

Data engineering

Azure Databricks

PySpark

SQL

Delta Lake

Python

Terraform

CI/CD

Azure DevOps

Git

Data governance

Data quality

Stakeholder collaboration

Education

Bachelor's degree in Computer Science or Information Systems

Tools

Terraform

Unity Catalog

Azure RBAC

Managed identities

Git

Azure DevOps

Azure Databricks

Job description

Location (City State, Street Name): Charlotte NC 28273

Type: Contract to Hire Only W2

• Large Healthcare organisation is seeking a lead Data Engineer to join their expanding team
• Must have very strong Data Bricks experience using the latest version
• Poweer BI and Microstrategy experience highly preferred
• Reporting platform will be Power Bi

This is What You’ll Do

• Lead design, development, and optimization of scalable data pipelines and ETL/ELT workflows using Azure Databricks, PySpark, SQL, and Delta Lake
• Enhancement of CI/CD pipelines and Azure DevOps, Git, and automated deployment frameworks to streamline development and release processes
• Ensure data quality and consistency through data integration, transformation, and cleansing
• Familiarity with security and governance best practices using Unity Catalog, Azure RBAC, managed identities, and data access controls to ensure compliance and secure data management is preferred
• Responsibility includes leading a team of data engineers and mentoring in best practices.
• Utilize Python for data manipulation, automation, and custom data solutions
• Collaborate with stakeholders to understand data needs and translate them into technical solutions
• Participate in on-call rotation and assist with after-hours issues, as required
• Perform other duties as assigned

This is What it Takes

• Bachelor’s degree in computer science or information systems science, or equivalent experience
• Three (3) years experience with Azure Databricks platform
• Five (5) years of comprehensive data warehouse analysis and design experience, with full knowledge of data warehouse methodologies, data modeling and familiarity with operational data stores.
• Five (5) years of experience with ETL/ELT Development, Python, PySpark
• Python, SQL, and Terraform experience

Required Certifications, Licenses, and Registrations

Knowledge, Skills, and Abilities

• Ability to adapt to changing priorities and respond to problems creatively
• Ability to assist clients with migrating from legacy to modern data platform
• Proficiency with technical, conceptual, interpersonal, written, and verbal communication skills
• Knowledge of general business, quality and regulatory requirements preferred
• Familiarity with Power BI and MicroStrategy architecture is a plus