Gcp Data Engineer

Nustar Technologies; Hyderabad, Pune District, Bengaluru; On-site; INR 1,400,000 - 2,200,000; Full time

Job summary

Nustar Technologies is seeking a GCP Data Engineer to design and manage data ingestion pipelines using our in-house Origin Data Product framework. You will work across GCS, BigQuery, Composer, Data Fusion, and Dataproc to ensure reliable ETL/ELT operations and mentor improvements in the framework.

The role calls for 5+ years of data engineering experience, strong scripting with Python, and IaC with Terraform.

Qualifications

• 5+ years of experience as a Data Engineer or in a similar role.
• Strong working knowledge of Google Cloud Storage (GCS), BigQuery, Composer, Data Fusion, Dataproc.
• Proficiency in Python for scripting, debugging, and automation tasks.
• Experience with Terraform for infrastructure-as-code.
• Knowledge of Git for version control.
• Understanding of data ingestion concepts, file formats, and ETL workflows.
• Ability to debug and resolve runtime data issues independently.
• Strong problem-solving and analytical skills.

Responsibilities

• Configure and execute data ingestion pipelines using our reusable GCP-based framework.
• Work with services such as GCS, BigQuery, Composer, Data Fusion, DataProc for ETL/ELT operations.
• Manage parameters, job configurations, and metadata for ingestion.
• Debug and resolve issues related to data, parameters, and job execution.
• Escalate framework-related bugs to the core development team when required.
• Monitor daily job runs, troubleshoot failures, and ensure SLAs are met.
• Collaborate with cross-functional teams for smooth delivery.
• Follow version control best practices using Git.
• Maintain deployment scripts and infrastructure configurations via Terraform.
• Suggest improvements to the framework or processes based on usage experience.
• Contribute to small enhancements in the framework, where applicable.

Skills

Python

ETL concepts

Data ingestion

Problem solving

Collaboration

Tools

GCS

BigQuery

Composer

Data Fusion

Dataproc

Terraform

Git

SQL

Job description

GCP Data Engineer

Experience: 712 Years
Location: Pune / Hyderabad / Chennai / Bangalore
Notice Period: Immediate to 30 Days

Role Overview:

We are looking for a Data Engineer with hands-on experience in Google Cloud Platform (GCP) services to work on implementing and managing data ingestion using our in-house Origin Data Product (ODP) framework.

The role primarily involves configuring pipelines, loading data, debugging issues, and ensuring smooth operation of the data ingestion process. Should be capable of handling parameter changes, data issues, and basic fixes within the scope of ingestion jobs. If you understand the framework well, you are encouraged to suggest improvements and may contribute to enhancements in collaboration with the core development team.

Key Responsibilities:

• Configure and execute data ingestion pipelines using our reusable GCP-based framework.
• Work with services such as GCS, BigQuery, Composer, Data Fusion, DataProc for ETL/ELT operations.
• Manage parameters, job configurations, and metadata for ingestion.
• Debug and resolve issues related to data, parameters, and job execution.
• Escalate framework-related bugs to the core development team when required.
• Monitor daily job runs, troubleshoot failures, and ensure SLAs are met.
• Collaborate with cross-functional teams for smooth delivery.
• Follow version control best practices using Git.
• Maintain deployment scripts and infrastructure configurations via Terraform.
• (Nice-to-have) Work with TWSd job scheduling/monitoring tools.
• Suggest improvements to the framework or processes based on usage experience.
• Contribute to small enhancements in the framework, where applicable.

Required Skills & Experience:

• 5+ years of experience as a Data Engineer or in a similar role.
• Strong working knowledge of Google Cloud Storage (GCS), BigQuery, Composer, Data Fusion, Dataproc.
• Proficiency in Python for scripting, debugging, and automation tasks.
• Experience with Terraform for infrastructure-as-code.
• Knowledge of Git for version control.
• Understanding of data ingestion concepts, file formats, and ETL workflows.
• Ability to debug and resolve runtime data issues independently.
• Strong problem-solving and analytical skills.

Good to Have:

• Exposure to TWSd or other enterprise job schedulers.
• Basic understanding of SQL optimization in BigQuery.

Soft Skills:

• Attention to detail and ownership mindset.
• Good communication and collaboration skills.
• Ability to work in a fast-paced environment and meet deadlines.
• Proactive approach towards process improvement and innovation.