Hybrid Data Engineer — Delta Lake, PySpark & Azure DevOps

BinaryBees Business Solutions LLC; Itasca (IL); Hybrid; USD 100,000 - 130,000; Full time

Job summary

A technology solutions provider in Illinois is seeking a Data Engineer to design, develop, and maintain scalable data pipelines. This mid-senior level position requires expertise in PySpark and Python, along with 5+ years of relevant experience. The role involves partnerships with data scientists, ensuring data quality, and optimizing performance. Applicants must possess a bachelor’s degree in Computer Science or a related field, along with strong skills in data integration and CI/CD pipelines.

Qualifications

• 5+ years of experience as a Data Engineer.
• 3+ years of experience with Delta Lake and change data feeds.
• Experience with data governance and security best practices.

Responsibilities

• Design and develop scalable data pipelines to process large volumes of data.
• Ensure data quality through validation and automated testing.
• Monitor and troubleshoot data pipeline issues.

Skills

Data pipeline development

PySpark

Python

CI/CD pipelines

Data lakehouse architecture

Object-Oriented Programming

Data integration

Education

Bachelor’s degree in Computer Science or related field

Tools

Azure DevOps

Databricks

Job description

A technology solutions provider in Illinois is seeking a Data Engineer to design, develop, and maintain scalable data pipelines. This mid-senior level position requires expertise in PySpark and Python, along with 5+ years of relevant experience. The role involves partnerships with data scientists, ensuring data quality, and optimizing performance. Applicants must possess a bachelor’s degree in Computer Science or a related field, along with strong skills in data integration and CI/CD pipelines.