GCP Data Engineer

Horizontal Talent; Bengaluru; Presencial; INR 900.000 - 1.350.000; Jornada completa

Descripción de la vacante

Horizontal Talent in Bengaluru is seeking a Data Engineer to design, build, and optimize data pipelines in a GCP environment, with a focus on Spark-based processing and BigQuery.

The role emphasizes hands-on development of Spark pipelines (Scala preferred), SQL in BigQuery, Unix shell scripting, and collaboration with analysts and stakeholders to ensure data quality and reliable analytics.

Formación

• 3-6 years of hands-on data engineering experience.
• Experience in GCP ecosystem including Dataproc, BigQuery, Cloud Storage.
• Proficient in SQL with BigQuery dialects and Unix/Linux scripting.
• Familiarity with Git and CI/CD processes.
• Strong problem-solving, debugging, and communication skills.

Responsabilidades

• Design, develop, and deploy data pipelines using Apache Spark (Scala).
• Work with GCP components: Dataproc, BigQuery, Cloud Storage.
• Write complex SQL queries for data analysis and transformation in BigQuery.
• Develop Unix shell scripts for automation and orchestration of data workflows.
• Optimize pipelines for performance, scalability, and reliability.
• Collaborate with engineers, analysts, and stakeholders to ensure data quality and meet requirements.

Conocimientos

Spark pipelines (Scala preferred)

BigQuery

Unix/Linux scripting

SQL (BigQuery dialects)

Git & CI/CD

Problem-solving

Communication & documentation

Herramientas

Dataproc

Cloud Storage

Descripción del empleo

We are seeking a highly skilled Data Engineer to support the development and optimization of data pipelines within our Google Cloud Platform (GCP) environment. The ideal candidate will have strong experience building Spark pipelines (Scala preferred), working with BigQuery, and writing efficient Unix scripts to support large-scale data processing.

You will collaborate with data engineering teams to design, build, and maintain high-performance data pipelines that enable analytics, reporting, and business insights.

Key Responsibilities

• Design, develop, and deploy data ingestion and transformation pipelines using Apache Spark (Scala).


• Work within Google Cloud Platform (GCP) components including Dataproc, Big Query & Cloud Storage.


• Write complex SQL queries for data analysis, validation, and transformation in Big Query.


• Develop Unix shell scripts for automation, data movement, and pipeline orchestration.


• Optimize and troubleshoot data pipelines for performance, scalability, and reliability.


• Collaborate with data engineers, analysts, and business stakeholders to understand requirements and ensure data quality.


• Contribute to code reviews, documentation, and CI/CD integration of data workflows.

Required Skills & Qualifications

• 3-6 years of hands-on experience in data engineering or related roles.


• Strong experience working in the Google Cloud Platform (GCP) ecosystem - including Dataproc, Big Query, Cloud Storage.


• Proficient in SQL (especially Big Query SQL dialects).


• Strong experience with Unix/Linux scripting for data automation.


• Familiarity with version control (Git) and CI/CD processes.


• Excellent problem-solving and debugging skills.


• Strong communication and documentation abilities.

Nice-to-Have Skills

• Experience with Python for data processing or automation.


• Knowledge of data governance, data quality, or metadata management best practices.