Senior Data Engineer - Google Cloud & Real-Time Pipelines

IBM; San Francisco (CA); Remote; USD 130,000 - 170,000; Full time

Job summary

IBM Consulting seeks a seasoned Data Engineer specializing in Google’s data platforms to design, build, and maintain data pipelines on Google’s Cloud ecosystem, leveraging open-source technologies for scalable data solutions.

Responsibilities include designing pipelines with DataProc, DataFlow, Pub/Sub, BigQuery, and BigTable; building data engineering solutions with Spark, Hadoop, Airflow, dbt; managing data platforms with Cloud Scheduler and Cloud Composer; optimizing data layers for migration

Qualifications

• Proven experience designing and maintaining data pipelines on Google Cloud.
• Strong experience with open-source tools such as Apache Beam, Airflow, dbt, Spark and Python/Scala.
• Experience with batch and real-time data integration for data warehouses and data lakes.
• Ability to design scalable data platforms and optimize data layers.

Responsibilities

• Design data pipelines (batch and real-time) for data warehouse and data lake on Google Cloud.
• Develop data engineering solutions using Google Cloud Storage, BigTable, BigQuery DataProc with Spark/Hadoop, DataFlow with Apache Beam or Python, and dbt.
• Manage data platforms with Google Cloud Scheduler and Cloud Composer (Airflow).
• Optimize data layer for efficient migration and processing on Google Cloud.
• Ensure scalability and efficiency of pipelines to meet business needs.

Skills

Google Data Platforms

Open-Source Technologies

Batch & Real-Time Pipelines

Data Platform Management

Data Layer Optimization

Tools

Apache Beam

Apache Airflow

dbt

Spark/Python

Spark/Scala

Google Cloud Services

Job description

IBM Consulting seeks a seasoned Data Engineer specializing in Google’s data platforms to design, build, and maintain data pipelines on Google’s Cloud ecosystem, leveraging open-source technologies for scalable data solutions.

Responsibilities include designing pipelines with DataProc, DataFlow, Pub/Sub, BigQuery, and BigTable; building data engineering solutions with Spark, Hadoop, Airflow, dbt; managing data platforms with Cloud Scheduler and Cloud Composer; optimizing data layers for migration