GCP Data Engineer

Compunnel, Inc.; Lisle (IL); On-site; USD 90,000 - 120,000; Full time

Job summary

Compunnel, Inc. is offering a Data Engineer position in Lisle, Illinois. This role requires 6+ years of experience in data engineering, particularly with Google Cloud Platform services. You will design and support data pipelines while mentoring junior engineers.

The ideal candidate should be an expert in SQL and Python, with a strong background in real-time data ingestion and familiarity with workflow management tools like Airflow. Join us to drive data solutions that enhance business decisions!

Qualifications

• 6+ years of experience in data engineering solutions.
• 2+ years of experience in Google Cloud Platform services.
• Expert knowledge of SQL and Python.

Responsibilities

• Collect business requirements by collaborating with various groups.
• Design, develop, deploy, and support high-performance data pipelines.
• Mentor and guide junior data engineers.

Skills

Data engineering solutions

SQL

Python programming

Google Cloud Platform (GCP)

Airflow

Real-time data ingestion

Job description

JOB SUMMARY

This role involves collaborating with business, IT, Analyst, and Data Science groups to define data requirements. The position requires designing, developing, deploying, and supporting high-performance data pipelines, modeling data platforms, and ensuring data quality. It also includes developing REST APIs for data exposure and mentoring junior data engineers.

Key Responsibilities

• Collect business requirements by working closely with various business, IT, Analyst, and Data Science groups.
• Design, develop, deploy, and support high-performance data pipelines, both inbound and outbound.
• Model the data platform by applying business logic and building objects in the semantic layer.
• Optimize data pipelines for performance, scalability, and reliability.
• Implement CI/CD pipelines for continuous deployment and delivery of data products.
• Ensure the quality of critical data elements, prepare data quality remediation plans, and collaborate to fix quality issues.
• Document the design and support strategy of data pipelines.
• Capture, store, and socialize data lineage and operational metadata.
• Troubleshoot and resolve data engineering issues.
• Develop REST APIs to expose data to other teams.
• Mentor and guide junior data engineers.

Required Qualifications

• 6+ years of experience in data engineering solutions such as data platforms, ingestion, data management, or publication/analytics.
• 2+ years of experience in Google Cloud Platform (GCP) with services like BigQuery, Composer, GCS, DataStream, or Dataflows.
• Expert knowledge of SQL and Python programming.
• Experience working with Airflow as a workflow management tool, including building operators to connect, extract, and ingest data.
• Experience in tuning queries for performance and scalability.
• Experience in real-time data ingestion using GCP Pub/Sub, Kafka, Spark, or similar technologies.
• Proven experience working in an agile environment.
• Proven experience working in incremental execution through successful launches.

Preferred Qualifications

• None specified in the provided text.

Certifications

• None specified in the provided text.