Data Engineer – Google Cloud Real-Time Pipelines (Remote)

IBM; Boston (MA); Hybrid; USD 215,000 - 253,000; Full time

Benefits offered by this job

Health benefits

401(k)

Paid time off

Training resources

Diversity groups

Job summary

IBM Consulting is seeking a seasoned Data Engineer to design, build, and maintain data pipelines on Google Cloud, leveraging DataProc, DataFlow, PubSub, BigQuery and BigTable. You will work with open-source technologies like Apache Beam, Airflow, and dbt to deliver scalable data solutions.

You will design end-to-end pipelines, manage data platforms, and optimize data layers for migration and processing. This role can be performed from anywhere in the United States.

Qualifications

• Deep Expertise in Google Data Platforms and cloud services.
• Experience with batch and real-time data pipelines using Google Cloud services.
• Ability to integrate open-source tools with Google Cloud platforms.

Responsibilities

• Design and build batch and real-time data pipelines for Data Warehouse and Datalake on Google Cloud.
• Develop and maintain data engineering solutions using Google Cloud Storage, BigQuery, BigTable and Spark/Python.
• Manage data platforms using Cloud Scheduler and Cloud Composer (Airflow).
• Optimize data layers for efficient migration and processing on Google Cloud services.
• Ensure scalability and reliability of data pipelines to meet business needs.

Skills

Google Data Platforms

Apache Beam

Apache Airflow

dbt

Spark

Python

BigQuery

BigTable

Cloud Spanner

CloudSQL

AlloyDB

Education

Bachelor's Degree

Tools

DataProc

DataFlow

PubSub

BigQuery

BigTable

Cloud Spanner

CloudSQL

AlloyDB

Apache Beam

Apache Airflow

Job description

IBM Consulting is seeking a seasoned Data Engineer to design, build, and maintain data pipelines on Google Cloud, leveraging DataProc, DataFlow, PubSub, BigQuery and BigTable. You will work with open-source technologies like Apache Beam, Airflow, and dbt to deliver scalable data solutions.

You will design end-to-end pipelines, manage data platforms, and optimize data layers for migration and processing. This role can be performed from anywhere in the United States.