Lead Data Engineer

Associate Staffing; Charlotte (NC); On-site; USD 130,000 - 140,000; Full time

Job summary

Associate Staffing is recruiting a Lead Data Engineer to design and optimize scalable data pipelines in a fast-growing plasma donation business. You will lead ETL/ELT workflows using Azure Databricks, PySpark, SQL and Delta Lake, mentor engineers, and drive data-driven decisions.

Based onsite in Charlotte, NC, you will collaborate with stakeholders, strengthen data governance with Unity Catalog and Azure RBAC, and advance CI/CD pipelines with Azure DevOps.

Qualifications

• Lead design, development, and optimization of scalable data pipelines and ETL/ELT workflows using Azure Databricks, PySpark, SQL, and Delta Lake.
• Drive CI/CD pipelines and automated deployment using Azure DevOps, Git, and automation frameworks.
• Ensure data quality, security, and governance with Unity Catalog, Azure RBAC, and managed identities.
• Lead and mentor a team of data engineers, promoting best practices and technical growth.
• Collaborate with stakeholders to translate data needs into technical solutions.

Responsibilities

• Lead the architecture and implementation of scalable data pipelines.
• Mentor and grow the data engineering team through code reviews and coaching.
• Collaborate with data scientists and analysts to enable data-driven decisions.
• Ensure data integrity, security, and governance across platforms.
• Support on-call duties and assist with incident resolution as needed.

Skills

Azure Databricks

PySpark

SQL

Data pipeline design

Leadership

Data governance

Education

Bachelor's degree in Computer Science or related field

Tools

Azure DevOps

Git

Automation frameworks

Unity Catalog

Azure RBAC

Job description

Lead Data Engineer
Target Salary: $130,000 - $140,000 per year

Join a dynamic organization dedicated to innovation and growth in the plasma donation industry. As a Lead Data Engineer, you will play a vital role in designing and optimizing scalable data pipelines that drive data-driven decisions and enhance operational efficiency. This position offers the opportunity to lead technical projects, mentor team members, and influence the future of data management within the company—without revealing specific locations or brands.

What You'll Do

• Lead the design, development, and optimization of scalable data pipelines and ETL/ELT workflows using Azure Databricks, PySpark, SQL, and Delta Lake
• Enhance CI/CD pipelines through Azure DevOps, Git, and automation frameworks to improve development and deployment processes
• Ensure data quality, consistency, and security through best practices in data integration, transformation, and governance using Unity Catalog, Azure RBAC, and managed identities
• Lead and mentor a team of data engineers, promoting best practices and technical growth
• Collaborate with stakeholders to translate data needs into technical solutions and participate in resolving complex data issues
• Support on-call responsibilities and assist with after-hours troubleshooting as needed

Top 5 Required Skills

• Expertise in Azure Databricks, PySpark, and SQL
• Proven experience in designing and managing scalable data pipelines and ETL/ELT workflows
• Strong knowledge of CI/CD processes, Azure DevOps, and automation frameworks
• Excellent communication and leadership skills, with team-leading experience considered a big plus
• Data security and governance knowledge, including Unity Catalog and Azure RBAC

Top 5 Nice to Have Skills

• Experience with Python for data manipulation and automation projects
• Knowledge of Delta Lake architecture and data management best practices
• Familiarity with Terraform for infrastructure as code
• Experience with Power BI, MicroStrategy, or similar BI tools
• Certifications such as Databricks Data Engineer Professional or Associate

Preferred Education and Experience

• Bachelor’s degree in Computer Science, Information Systems, or a related field
• Minimum of 3 years of experience with Azure Databricks
• At least 5 years in data warehouse analysis, design, and ETL/ELT development
• Extensive experience with data modeling, operational data stores, and data governance

Other Requirements

• Ability to work onsite at our corporate location in Charlotte, NC
• Capable of sitting, standing, and lifting up to 20 pounds occasionally
• Excellent verbal and written communication skills
• Willing to travel up to 20% for business needs
• Ability to use assistive devices, if necessary, for mobility or communication

Associate Staffing is an Equal Opportunity Employer and does not discriminate against any employee or applicant for employment based on race, color, religion, sex, national origin, age, disability, veteran status, sexual orientation, gender identity, or any other protected status under applicable law. We are committed to creating a diverse and inclusive work environment and welcome applicants from all backgrounds to apply for open positions with our company.