Lead Data Engineer - Google Cloud Platform

Deqode; Gurugram District; On-site; INR 80,000 - 100,000; Full time

Job summary

Deqode is seeking a skilled professional to implement and architect solutions on Google Cloud Platform. Candidates should have expertise in designing scalable ETL/ELT pipelines using PySpark and experience with Apache Beam or Apache Spark.

A Bachelor's or Master's degree in a related field is required, along with strong proficiency in Python and SQL. Relevant certifications in Google Cloud will be an advantage. The position is based in Gurugram District, Haryana, India.

Qualifications

• Experience with creating end-to-end data pipelines using Apache Beam, Google Dataflow, or Apache Spark.
• Expertise in at least two database technologies: Relational, Analytical, or NoSQL databases.
• Strong proficiency in Python and SQL essential.

Responsibilities

• Implement and architect solutions on Google Cloud Platform using GCP components.
• Design, develop, and deploy scalable ETL/ELT pipelines using PySpark on GCP.
• Optimize jobs for performance and reliability.

Skills

Google Cloud Platform

Apache Beam

Python

SQL

PySpark

Data warehousing

Apache Airflow

Education

Bachelor's or Master's degree in Computer Science, Engineering, or a related field

Tools

Google BigQuery

Hadoop

Job description

Job Requirements

• Have Implemented and Architected solutions on Google Cloud Platform using the components of GCP
• Experience with Apache Beam/Google Dataflow/Apache Spark in creating end to end data pipelines.
• Experience in some of the following : Big Query, Big Table Cloud Storage, Datastore, Spanner, Cloud SQL, Machine Learning.
• Experience programming in Hadoop, python, SQL
• Expertise in at least two of these technologies: Relational Databases, Analytical Databases, NoSQL databases.
• Certified in Google Professional Data Engineer/ Solution Architect is a major Advantage

Skills Required

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
• Design, develop, and deploy scalable ETL/ELT pipelines using PySpark on GCP.
• Utilize GCP services extensively, including BigQuery (data warehousing), Cloud Storage, Dataproc, and Dataflow.
• Optimize PySpark jobs for performance and reliability, and fine-tune BigQuery queries.
• Implement complex transformations and process large volumes of structured/unstructured data using Spark SQL and PySpark
• Build and manage automated workflows using Apache Airflow or Cloud Composer.
• Strong proficiency in Python and SQL is essential
• Proven experience with Google Cloud Platform (GCP) services.
• Relevant certifications on google cloud is an added advantage.