Gcp Developer

Cloudxtreme; Hyderabad, Pune District, Bengaluru; On-site; INR 1,200,000 - 2,200,000; Full time

Job summary

Cloudxtreme Hyderabad, India, is seeking a data engineering professional to analyze data and design complex SQL queries in GCP BigQuery. You will build stream and batch data pipelines using Java with Apache Beam on Google Dataflow, orchestrate with Airflow/Cloud Composer, and manage streaming inputs via Pub/Sub.

You will deploy infrastructure with Terraform, store code in Bitbucket, and document work in JIRA/Confluence, while collaborating with BSA-driven requirements.

Qualifications

• Analyze data and write complex SQL queries in GCP BigQuery.
• Strong SQL skills for analyses and aggregations.
• ETL concepts knowledge.
• Experience with Java and Python for data pipelines.
• Experience building data pipelines with Apache Beam and Google Dataflow.
• Knowledge of Airflow/Cloud Composer.
• Familiarity with Pub/Sub streaming data.
• Terraform for deployment in cloud environments.
• Version control with Bitbucket, Jira, and Confluence usage.
• Basic Python programming knowledge.

Responsibilities

• Analyze data and write complex SQL queries in GCP BigQuery to extract required information.
• Develop and maintain data pipelines using Java/Python.
• Build stream and batch data pipelines with Apache Beam and Dataflow.
• Orchestrate workloads with Airflow/Cloud Composer.
• Receive streaming data via Pub/Sub and process it.
• Maintain code in Bitbucket and deploy using Terraform.
• Document results and update tasks in JIRA/Confluence.
• Collaborate with stakeholders to understand requirements.

Skills

SQL

ETL concepts

Java

Python

Apache Beam

Google Dataflow

Airflow

Cloud Composer

Pub/Sub

Terraform

Bitbucket

JIRA

Confluence

Requirements gathering

Tools

BigQuery

Google Dataflow

Apache Beam

Terraform

Bitbucket

JIRA

Confluence

Pub/Sub

Job description

Role & responsibilities

• Analyze data and write complex SQL queries in GCP Big Query to extract required information.
• Expertise in ETL concepts
• Very strong SQL, ability to write complex SQL to perform common types of analysis and aggregations
• Create stream and batch data pipeline using Java in Apache beam with distributed data processing using Google Dataflow.
• Expert in Airflow, Cloud Composer, Dataflow, Cloud Storage balancing
• Understand customer requirement lead down by BSA and provide required data outputs.
• Orchestrate data pipeline jobs using Google cloud composer.
• Write code in Python / Java to receive streaming data from Google cloud pub/sub.
• Maintain code in given Bit bucket code repository and use Terraform and Python based data pipeline to deploy code in various cloud environment.
• Accept work, update status in JIRA and create documentation in Confluence.
• Basic knowledge in Python programming,