Google Cloud Data Engineer - Pipelines & Platforms

International Business Machines Corporation; New York (NY); Remote; USD 130,000 - 190,000; Full time

Job summary

IBM Consulting invites a seasoned Data Engineer to design, build, and maintain data solutions on Google's Cloud ecosystem. You will leverage DataProc, DataFlow, PubSub, BigQuery, BigTable, and open-source tech to deliver scalable pipelines across data warehouse and data lake environments.

You'll work with a global team, integrating Beam, Airflow, dbt, Spark, and Python to create robust data platforms. The role can be performed from anywhere in the United States, with collaboration across

Qualifications

• Proven experience designing, building, and maintaining data engineering solutions on Google Cloud.
• Experience with batch and real-time data pipelines for data warehouse and data lake.
• Ability to integrate open-source tools with Google Cloud services.
• Experience scheduling and managing data platforms.
• Knowledge of data modeling and scalable data architectures.

Responsibilities

• Design batch and real-time data pipelines using Google Cloud.
• Build and maintain data engineering solutions with Spark, Python, and dbt.
• Manage data platforms with Cloud Composer and Cloud Scheduler.
• Optimize data layers for migration and processing.
• Ensure scalability of data pipelines to meet business needs.

Skills

Google Cloud Platforms

Data Pipelines

Batch & Real-Time

Open-Source Tools

Python

Spark

Scala

DBT

Airflow

BigQuery

Tools

Google DataProc

DataFlow

Pub/Sub

BigQuery

BigTable

Cloud Spanner

CloudSQL

AlloyDB

Apache Beam

Airflow

dbt

Spark

Python

Spark/Scala

Cloud Scheduler

Cloud Composer

Job description

IBM Consulting invites a seasoned Data Engineer to design, build, and maintain data solutions on Google's Cloud ecosystem. You will leverage DataProc, DataFlow, PubSub, BigQuery, BigTable, and open-source tech to deliver scalable pipelines across data warehouse and data lake environments.

You'll work with a global team, integrating Beam, Airflow, dbt, Spark, and Python to create robust data platforms. The role can be performed from anywhere in the United States, with collaboration across