Senior GCP Data Engineer: AI-Driven Real-Time Pipelines

Ethereum Technologies; Hartford (CT); On-site; USD 120,000 - 180,000; Full time

Job summary

Ethereum Technologies is seeking an experienced data engineer to design and implement scalable data pipelines on Google Cloud Platform. You will develop workflows with Cloud Composer, write Python code, and optimize BigQuery SQL for large-scale data processing.

The role requires hands-on experience with real-time data streams (Kafka), NoSQL (MongoDB), and collaborating across onshore/offshore teams in a fast-paced environment.

Qualifications

• Design and implement robust data pipelines on GCP using BigQuery, Cloud Storage, and Pub/Sub.
• Develop and manage workflows with Cloud Composer for scheduling and orchestration.
• Write clean, scalable Python code using advanced techniques.
• Craft complex SQL queries in BigQuery with window functions, CTEs, and performance tuning.
• Build real-time data processing using Apache Kafka.
• Model and manage MongoDB NoSQL databases with scalable schemas.
• Utilize Shell scripting and Linux administration to support data infra.
• Perform data profiling and implement validation for data quality.
• Develop API integration scripts for seamless service automation.
• Create data flow diagrams to communicate architecture and pipeline logic.

Responsibilities

• Design and implement robust data pipelines on GCP.
• Develop and manage workflows using Cloud Composer for orchestration.
• Write Python code for data processing and automation.
• Craft complex BigQuery SQL for performance tuning.
• Build real-time data processing systems with Kafka.
• Maintain MongoDB schemas for scalable data models.
• Provide automation scripts to integrate services and data sources.
• Collaborate with onshore/offshore teams to deliver pipelines and support.

Skills

AI/GPT experience

Gemini experience

GCP data services

Python

SQL (BigQuery)

Apache Kafka

NoSQL MongoDB

Shell scripting

Linux administration

Data profiling/quality

API integration

Data pipelines

Cloud Composer (Airflow)

Education

GCP Google Cloud Professional Data Engineer Certification

Tools

GitHub

RTC

Automation tools

Cloud Composer

Job description

Ethereum Technologies is seeking an experienced data engineer to design and implement scalable data pipelines on Google Cloud Platform. You will develop workflows with Cloud Composer, write Python code, and optimize BigQuery SQL for large-scale data processing.

The role requires hands-on experience with real-time data streams (Kafka), NoSQL (MongoDB), and collaborating across onshore/offshore teams in a fast-paced environment.