GCP Data Engineer

Acme Services; Gurugram District; On-site; INR 1,200,000 - 2,200,000; Full time

Job summary

Acme Services in Gurugram is seeking a Data Engineer to design, build and maintain scalable data pipelines on Google Cloud Platform. You will work with cross-functional teams to deliver reliable data solutions for analytics. The role focuses on ETL/ELT workflows, SQL modeling in BigQuery, and both batch and streaming processing using Pub/Sub and Dataflow.

Proficiency in Python and PySpark is required. Join a collaborative environment that emphasizes data governance, cost optimization, and

Responsibilities

• Design, develop and maintain scalable data pipelines on Google Cloud Platform.
• Build ETL/ELT workflows using Cloud Dataflow, Dataproc, Cloud Composer and BigQuery.
• Develop and optimize SQL queries and data models in BigQuery.
• Ingest data from databases, APIs, files and streaming systems.
• Work with Python, PySpark and SQL for data processing and transformation.
• Implement batch and real-time data processing using Pub/Sub and Dataflow.
• Design data warehouses/data lakes using BigQuery, Cloud Storage and Dataplex.
• Ensure data quality, accuracy, security, governance and compliance.
• Optimize pipeline performance and GCP resource/cost utilization.
• Monitor, troubleshoot and resolve issues in production pipelines.
• Collaborate with Data Scientists, BI teams, Architects and stakeholders.
• Implement CI/CD, version control, testing and deployment for data engineering.
• Create and maintain technical documentation for pipelines, architecture and data models.

Job description

Role & responsibilities

• Design, develop, and maintain scalable data pipelines on Google Cloud Platform.
• Build ETL/ELT workflows using tools such as Cloud Dataflow, Dataproc, Cloud Composer, and BigQuery.
• Develop and optimize SQL queries and data models in BigQuery.
• Perform data ingestion from various sources including databases, APIs, files, and streaming systems.
• Work with Python, PySpark, and SQL for data processing and transformation.
• Implement batch and real-time data processing solutions using technologies such as Pub/Sub and Dataflow.
• Design data warehouses/data lakes using GCP services such as BigQuery, Cloud Storage, and Dataplex.
• Ensure data quality, accuracy, security, governance, and compliance.
• Optimize pipeline performance and GCP resource/cost utilization.
• Monitor, troubleshoot, and resolve issues in production data pipelines.
• Collaborate with Data Scientists, BI teams, Architects, and business stakeholders to understand data requirements.
• Implement CI/CD, version control, testing, and deployment practices for data engineering solutions.
• Create and maintain technical documentation for data pipelines, architecture, and data models.