Senior Google Cloud Data Engineer – Real-Time Pipelines

IBM; Boston (MA); Hybrid; USD 215,000 - 253,000; Full time

Benefits offered by this job

Healthcare benefits

401(k) plan

Paid time off

Job summary

IBM Consulting is seeking a seasoned Data Engineer for Google Cloud data platforms. You will design, build, and maintain scalable data pipelines across batch and real-time workloads, leveraging DataProc, DataFlow, PubSub, BigQuery, and Big Table.

You’ll optimize data layers and oversee data platform operations using Cloud Composer (Airflow) and Cloud Scheduler. The role emphasizes collaboration with teams to deliver robust data engineering solutions, requiring expertise in Spark/Python/Scala,

Qualifications

• Experience designing, building, and maintaining data pipelines on Google Cloud.
• Proficient with Beam, Airflow, dbt; able to integrate with Google Cloud services.
• Experience with batch and real-time data pipelines for Data Warehouse and Data Lake.
• Experience scheduling data platforms with Cloud Scheduler and Cloud Composer (Airflow).
• Bachelor's degree required.

Responsibilities

• Design and develop batch and real-time data pipelines for Data Warehouse and Datalake using Google Cloud services.
• Build and maintain data engineering solutions using Google Cloud Storage, BigQuery, and related tools.
• Manage data platforms: schedule and operate pipelines with Cloud Scheduler and Cloud Composer (Airflow).
• Design and optimize data layers for efficient data migration and processing.
• Ensure scalability and efficiency of data pipelines to meet business needs.

Skills

Google Cloud data platforms

Data pipeline design

Open-source tools

Batch processing

Real-time processing

Education

Bachelor's degree

Tools

Apache Beam

Apache Airflow

dbt

Spark (Python/Scala)

Google Cloud DataProc

BigQuery

PubSub

Cloud Composer

Job description

IBM Consulting is seeking a seasoned Data Engineer for Google Cloud data platforms. You will design, build, and maintain scalable data pipelines across batch and real-time workloads, leveraging DataProc, DataFlow, PubSub, BigQuery, and Big Table.

You’ll optimize data layers and oversee data platform operations using Cloud Composer (Airflow) and Cloud Scheduler. The role emphasizes collaboration with teams to deliver robust data engineering solutions, requiring expertise in Spark/Python/Scala,