Google Cloud Data Engineer

Cloudnyx.ai; Pune District; On-site; INR 1,800,000 - 2,800,000; Full time

Job summary

Cloudnyx.ai in Pune, India seeks a seasoned Data Engineer to design and implement scalable data pipelines and architectures for Google products and services. You will develop data models, schemas, and ontologies to support diverse data sources and collaborate with product managers, engineers, and researchers to translate data requirements into technical solutions.

You will build and optimize batch and real-time pipelines using GCP services (BigQuery, Dataflow, Dataproc, Pub/Sub, Cloud Storage),

Qualifications

• Bachelors or Masters in Computer Science, Engineering, or related field.
• 5+ years of experience in data engineering or a related field.
• Strong understanding of data warehousing, data modeling, and ETL processes.
• Expertise in designing and implementing large-scale data pipelines and architectures.
• Proficiency in SQL and at least one programming language such as Python or Java.
• Experience with Google Cloud Platform (GCP) services such as BigQuery, Dataflow, Dataproc, Pub/Sub, and Cloud Storage.
• Experience with open-source data processing frameworks such as Hadoop, Spark, and Kafka.
• Excellent communication, interpersonal, and collaboration skills.

Responsibilities

• Design and implement scalable data pipelines and architectures.
• Develop and maintain data models and schemas.
• Evaluate new data technologies to improve infrastructure.
• Collaborate with product managers, engineers, researchers to define data requirements.
• Build and optimize batch and real-time pipelines using GCP data services.
• Provide technical leadership and mentor junior engineers.

Skills

Data modeling

ETL processes

SQL

Python

Java

Data warehousing

Data governance

Education

Bachelor's or Master's degree in Computer Science, Engineering, or a related field

Tools

BigQuery

Dataflow

Dataproc

Pub/Sub

Cloud Storage

Spanner

Hadoop

Spark

Kafka

Job description

Responsibilities

• Design and implement scalable, reliable, and efficient data pipelines and architectures for various Google products and services.
• Develop and maintain data models, schemas, and ontologies to support diverse data sources and use cases.
• Evaluate and recommend new and emerging data technologies and tools to improve Google's data infrastructure.
• Collaborate with product managers, engineers, and researchers to define data requirements and translate them into technical solutions.

Data Processing and Pipelines

• Build and optimize batch and real-time data pipelines using Google Cloud Platform (GCP) services such as Dataflow, Dataproc, Pub/Sub, and Cloud Functions.
• Develop and implement data quality checks and validation processes to ensure data accuracy and consistency.
• Design and implement data governance policies and procedures to ensure data security and compliance.

Data Storage and Management

• Design and implement scalable data storage solutions using GCP services such as BigQuery, Cloud Storage, and Spanner.
• Optimize data storage and retrieval for performance and cost-effectiveness.
• Implement data lifecycle management policies and procedures.

Team Leadership and Mentorship

• Provide technical leadership and guidance to data engineers and other team members.
• Mentor and coach junior engineers to develop their skills and expertise.
• Foster a culture of innovation and collaboration within the team.

Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
• 5+ years of experience in data engineering or a related field.
• Strong understanding of data warehousing, data modeling, and ETL processes.
• Expertise in designing and implementing large-scale data pipelines and architectures.
• Proficiency in SQL and at least one programming language such as Python or Java.
• Experience with Google Cloud Platform (GCP) services such as BigQuery, Dataflow, Dataproc, Pub/Sub, and Cloud Storage.
• Experience with open-source data processing frameworks such as Hadoop, Spark, and Kafka.
• Excellent communication, interpersonal, and collaboration skills.

Preferred Qualifications

• Experience with data governance and data quality management.
• Experience with machine learning and data science.
• Experience with containerization and orchestration technologies such as Docker and Kubernetes.
• Contributions to open-source projects or communities.