Senior Data Engineer

The Planet Group; Irving (TX); Hybrid; USD 96,000 - 110,000; Full time

Job summary

Planet Group is partnering with a Chicago-area firm to hire a Data Engineer who will design, build, and maintain scalable ETL pipelines powering BI, analytics, and downstream apps. The role emphasizes Python, Spark, and SQL skills, data quality, and production-grade solutions.

You will collaborate with technical and business stakeholders, translate requirements into efficient workflows, and help improve pipeline performance in a hybrid Chicago environment with on-site three days a week.

Qualifications

• 5+ years of hands-on experience developing and maintaining ETL pipelines in production environments.
• Strong Python programming skills for data processing, transformation, and automation.
• Practical experience using Apache Spark to process large datasets.
• Advanced SQL skills, including complex queries and data transformations.
• Solid understanding of data engineering principles, including scalability, reliability, data quality, and performance optimization.
• Experience troubleshooting production data workflows and resolving technical issues.
• Strong communication, problem-solving, and collaboration skills.

Responsibilities

• Design, build, and maintain production ETL pipelines that extract, transform, and integrate large datasets.
• Develop data processing workflows using Python and Apache Spark.
• Write and optimize complex SQL queries to support data transformation, validation, and analysis.
• Partner with cross-functional teams to understand source data, define requirements, and deliver usable datasets.
• Implement data quality checks to improve accuracy and consistency across pipelines.
• Investigate pipeline failures, troubleshoot processing issues, and resolve performance bottlenecks.
• Improve the scalability, reliability, and maintainability of existing data solutions.
• Follow engineering best practices for version control, code documentation, and collaborative development.
• Participate in an agile team environment and communicate progress, issues, and technical decisions clearly.

Skills

Python

Apache Spark

SQL

ETL pipelines

Data quality

Troubleshooting

Communication

Collaboration

Tools

Azure Data Factory

Databricks

Power BI

Pulumi

Job description

Data Engineer

Location: Chicago

Hybrid role- candidates must be willing to go onsite three days a week

Rate: $70-$80/hour

H1b's are permitted but MUST be willing to sign a 90 day release.

The Planet Group has partnered with a Chicago area firm to locate a Data Engineer to build and support scalable data pipelines that power business intelligence, analytics, and downstream applications.

This hands-on role is ideal for someone with strong Python, Apache Spark, and SQL skills who enjoys solving complex data challenges and delivering reliable production solutions.

You will collaborate with technical and business partners to translate data requirements into efficient workflows, maintain data quality, and improve pipeline performance.

Key Responsibilities

• Design, build, and maintain production ETL pipelines that extract, transform, and integrate large datasets.
• Develop data processing workflows using Python and Apache Spark.
• Write and optimize complex SQL queries to support data transformation, validation, and analysis.
• Partner with cross-functional teams to understand source data, define requirements, and deliver usable datasets.
• Implement data quality checks to improve accuracy and consistency across pipelines.
• Investigate pipeline failures, troubleshoot processing issues, and resolve performance bottlenecks.
• Improve the scalability, reliability, and maintainability of existing data solutions.
• Follow engineering best practices for version control, code documentation, and collaborative development.
• Participate in an agile team environment and communicate progress, issues, and technical decisions clearly.

Required Qualifications

• 5+ years of hands‑on experience developing and maintaining ETL pipelines in production environments.
• Strong Python programming skills for data processing, transformation, and automation.
• Practical experience using Apache Spark to process large datasets.
• Advanced SQL skills, including complex queries and data transformations.
• Solid understanding of data engineering principles, including scalability, reliability, data quality, and performance optimization.
• Experience troubleshooting production data workflows and resolving technical issues.
• Strong communication, problem‑solving, and collaboration skills.

Preferred Qualifications

• Experience with Microsoft Azure data services, particularly Azure Data Factory.
• Experience developing or running Spark workloads in Databricks.
• Familiarity with Power BI and how data pipelines support reporting and analytics.
• Exposure to infrastructure as code tools, such as Pulumi.