Data Engineer II

Jobtailor; Illinois; On-site; USD 90,000 - 130,000; Full time

Job summary

Jobtailor in Illinois is seeking a Data Engineer to design, build, and maintain scalable ELT/ETL pipelines using Databricks, Apache Spark, SQL, and Python. You will develop high-quality datasets for analytics, reports, and AI use cases, and collaborate with stakeholders to translate data needs into robust solutions.

The role emphasizes data quality, modern cloud-native architectures, and best practices in governance, testing, and automation.

Qualifications

• Bachelor's degree in CS/IS/Engineering or equivalent practical experience.
• 3+ years of experience in Data Engineering or related role.
• Strong SQL development skills.
• Strong Python programming experience.
• Experience building data pipelines with Apache Spark and Databricks.
• Experience developing modern ETL/ELT pipelines.
• Understanding of data modeling and data warehousing concepts.
• Experience with cloud-based data platforms, preferably AWS.
• Familiarity with Delta Lake or similar storage technologies.
• Experience using Git and collaborative software development practices.
• Strong analytical, troubleshooting, and problem-solving skills.
• Excellent communication and collaboration skills.

Responsibilities

• Design, develop, and maintain scalable ELT/ETL pipelines using Databricks, Apache Spark, SQL, and Python.
• Build and maintain reliable datasets for analytics, reporting, and AI use cases.
• Develop data transformations using Delta Lake and Lakehouse best practices.
• Integrate data from enterprise applications, APIs, databases, files, and other sources.
• Collaborate with stakeholders to understand data requirements and deliver scalable solutions.
• Monitor, troubleshoot, and optimize production data pipelines for reliability and performance.
• Participate in migrating legacy processes to cloud-native architectures.
• Improve data quality through validation, testing, monitoring, and automation.
• Contribute to engineering best practices via code reviews and documentation.
• Support continuous improvement of the data platform by identifying automation opportunities.

Skills

SQL development

Python programming

Spark & Databricks

ETL/ELT pipelines

Delta Lake

Data warehousing

AWS

Git

Data modeling

Analytical skills

Communication

Education

Bachelor's degree in Computer Science / Information Systems / Engineering

Tools

Databricks

Delta Lake

AWS

Job description

Responsibilities

• Design, develop, and maintain scalable ELT/ETL pipelines using Databricks, Apache Spark, SQL, and Python
• Build and maintain reliable, high-quality datasets that support enterprise analytics, operational reporting, and AI use cases
• Develop data transformations using Delta Lake and Lakehouse architecture best practices
• Integrate data from enterprise applications, APIs, databases, files, and other internal and external sources
• Collaborate with business stakeholders, analysts, and product teams to understand data requirements and deliver scalable solutions
• Monitor, troubleshoot, and optimize production data pipelines for reliability, scalability, and performance
• Participate in migrating and modernizing legacy data processes to cloud-native architectures
• Improve data quality through validation, testing, monitoring, and automation
• Contribute to engineering best practices through code reviews, documentation, and knowledge sharing
• Support continuous improvement of the enterprise data platform by identifying opportunities for automation, optimization, and simplification

Requirements

• Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field, or equivalent practical experience
• 3+ years of experience in Data Engineering or a related technical role
• Strong SQL development skills
• Strong Python programming experience
• Experience building data pipelines using Apache Spark and Databricks (or similar distributed data processing platforms)
• Experience developing modern ETL/ELT pipelines
• Understanding of data modeling and data warehousing concepts
• Experience working with cloud-based data platforms, preferably AWS
• Familiarity with Delta Lake or similar modern data storage technologies
• Experience using Git and collaborative software development practices
• Strong analytical, troubleshooting, and problem-solving skills
• Excellent communication and collaboration skills

Core Competencies

Demonstrates expertise in designing and developing scalable ELT/ETL pipelines using Databricks, Apache Spark, SQL, and Python, while ensuring data quality and optimizing performance in cloud-based environments. Collaborates effectively with stakeholders to deliver data solutions that support enterprise analytics and operational reporting.