Data Engineer

Compunnel, Inc.; Lisle (IL); On-site; USD 100,000 - 130,000; Full time

Job summary

A data solutions company in Lisle seeks a Data Engineer to collaborate with teams on building scalable data solutions. You will design robust data pipelines and ensure data quality while leveraging Google Cloud services. Candidates should have significant experience in data engineering, proficiency in SQL and Python, along with a Bachelor's degree in a related field. The role offers opportunities for mentorship and working in an agile environment.

Qualifications

• 6+ years of experience in data engineering, including data platforms and analytics.
• Excellent problem-solving and critical-thinking abilities.
• Experience working in an agile development environment.

Responsibilities

• Collaborate with teams to gather and understand requirements.
• Design, develop, and maintain data pipelines.
• Optimize data pipelines for performance and reliability.
• Implement CI/CD pipelines for data products.
• Ensure data quality and document pipeline design.

Skills

SQL

Python

Data ingestion technologies (GCP Pub/Sub, Kafka, Spark)

Data modeling and semantics

CI/CD pipeline implementation

Airflow for workflow management

Query tuning

Agile development experience

Education

Bachelor's degree in Computer Science or related field

Master's degree preferred

Tools

Google Cloud services (BigQuery, Composer, GCS, DataStream, Dataflow)

Job description

The Data Engineer will collaborate with cross-functional teams including business stakeholders, IT, analysts, and data scientists to design and implement robust data pipelines.

This role involves building scalable data solutions, ensuring data quality, and supporting the organization's data infrastructure to enable efficient data access and analytics.

Key Responsibilities

• Collaborate with business, IT, analyst, and data science teams to gather and understand requirements.
• Design, develop, deploy, and maintain high-performance inbound and outbound data pipelines.
• Model data platforms by applying business logic and building semantic layer objects.
• Optimize data pipelines for performance, scalability, and reliability.
• Implement CI/CD pipelines for continuous deployment and delivery of data products.
• Ensure data quality by identifying critical data elements and preparing remediation plans.
• Document pipeline design and support strategies.
• Capture, store, and share data lineage and operational metadata.
• Troubleshoot and resolve data engineering issues.
• Develop REST APIs to expose data to internal teams.
• Mentor and guide junior data engineers.
• Perform other duties as assigned.
• Comply with all company policies and standards.

Required Qualifications

• Bachelor’s degree in Computer Science, Computer Engineering, Software Engineering, or a related technical field.
• Minimum 6 years of experience in data engineering, including data platforms, ingestion, data management, and analytics.
• At least 2 years of experience with Google Cloud services such as BigQuery, Composer, GCS, DataStream, and Dataflow.
• Proficiency in SQL and Python programming.
• Experience with Airflow for workflow management and custom operator development.
• Skilled in query tuning for performance and scalability.
• Experience with real-time data ingestion using GCP Pub/Sub, Kafka, Spark, or similar technologies.
• Strong organizational, prioritization, and analytical skills.
• Proven experience in incremental execution and successful product launches.
• Excellent problem-solving and critical-thinking abilities.
• Experience working in an agile development environment.

Preferred Qualifications

• Master’s degree in Computer Science, Computer Engineering, Software Engineering, or a related technical field.