GCP Data Engineer

Elabs Infotech; Khordha, Kolkata District; On-site; INR 4,000,000 - 7,000,000; Full time

Job summary

Elabs Infotech is hiring a Senior GCP Data Engineer / Technical Lead with deep hands-on experience in Dataflow, Apache Beam, IAM strategy, Dataproc, BigQuery, Cloud Composer, Python and SQL. You will design, develop and optimize scalable enterprise data platforms for leading clients.

Role involves architecting end-to-end data pipelines, building reusable templates, and mentoring peers to ensure performance, reliability, and cost efficiency across cloud-based data ecosystems.

Qualifications

• 4–8 years of hands-on data engineering experience.
• Strong experience with GCP Dataflow, Apache Beam, and Dataproc.
• Deep knowledge of BigQuery performance tuning and cost optimization.
• Proficiency in Python and SQL for large-scale data processing.

Responsibilities

• Design and implement complex, scalable data pipelines using GCP Dataflow, Apache Beam and Dataproc/Spark.
• Develop custom Dataflow/Beam templates and reusable pipeline frameworks.
• Design and optimize Cloud Composer/Airflow DAGs, dependencies, scheduling and failure/retry strategies.
• Develop advanced Python and GoogleSQL solutions for large-scale data processing.
• Perform BigQuery performance and cost optimization, including query tuning, partitioning and clustering.
• Troubleshoot complex pipeline failures, performance bottlenecks and production issues.
• Implement batch and streaming pipelines using Dataflow and Pub/Sub.
• Review code and technical designs, ensure engineering best practices and mentor team members.

Skills

GCP Dataflow

Apache Beam

Dataproc

BigQuery

Python

SQL

Cloud Composer

Airflow

Job description

Exp: 4 - 8 Years

Loc: Pan India (Preferred Kolkata)

Job Type: Full Time with CMMI Level 5 Company

Looking for immediate joiners only

We are hiring for one of our Leading Clients for a strong Senior GCP Data Engineer / Technical Lead with deep hands-on experience in Dataflow, Apache Beam, IAM strategy, Dataproc, BigQuery, Cloud Composer, Python and SQL to design, develop and optimize scalable enterprise data platforms.

Key Responsibilities

• Design and implement complex, scalable data pipelines using GCP Dataflow, Apache Beam and Dataproc/Spark.
• Develop custom Dataflow/Beam templates and reusable pipeline frameworks.
• Design and optimize Cloud Composer/Airflow DAGs, dependencies, scheduling and failure/retry strategies.
• Develop advanced Python and GoogleSQL solutions for large-scale data processing.
• Perform BigQuery performance and cost optimization, including query tuning, partitioning and clustering.
• Troubleshoot complex pipeline failures, performance bottlenecks and production issues.
• Implement batch and streaming pipelines using Dataflow and Pub/Sub.
• Review code and technical designs, ensure engineering best practices and mentor team members.
• Own technical quality across modules covering performance, scalability, reliability and maintainability.
• Collaborate with architects and stakeholders to translate solution designs into robust technical implementations.

Mandatory Skills

GCP | Dataflow | Apache Beam | Dataproc | Spark/PySpark | BigQuery | Cloud Composer | Airflow | Python | GoogleSQL/SQL

• Strong hands-on Dataflow & Apache Beam experience is mandatory.
• Strong knowledge of GCP IAM, service accounts and project-level security.
• Expertise in BigQuery optimization, partitioning, clustering and cost management.
• Strong understanding of Dataproc/Spark internals and performance tuning.
• Hands-on experience with Cloud Composer/Airflow and production-grade DAG development.
• Strong troubleshooting and technical problem-solving skills.