Data Engineer – GCP Java & Big Data

United States Digital Space LLC; Karnataka; On-site; INR 2,500,000 - 4,500,000; Full time

Job summary

United States Digital Space LLC in Gurgaon/Bengaluru is seeking a GCP Data Engineer with 6–9 years of hands-on experience in building scalable data solutions using Java and Google Cloud Platform.

The role focuses on ETL/ELT pipelines, batch and streaming processing with Dataproc, Dataflow, Spark, and BigQuery, with strong SQL and data warehousing skills. Experience with Airflow/Cloud Composer and cloud migrations is a plus.

Qualifications

• Proficiency in Java with experience building distributed data processing applications.
• Hands-on with Spark, Hadoop, and Hive in big data environments.
• Experience with Spark DataFrame/SQL using Java or Scala; PySpark is a plus.
• Strong SQL, data modelling and data warehousing knowledge.
• Experience with Linux/Unix and scripting (Bash or similar).
• Analytical and problem-solving skills for debugging data pipelines.
• Ability to design scalable, fault-tolerant data processing systems.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Java-based big data frameworks on GCP.
• Build and optimize batch and streaming data processing with Dataproc, Dataflow, and Spark.
• Ensure high-quality, maintainable code following best practices and standards.
• Perform unit and integration testing and troubleshoot complex data pipelines.
• Collaborate with cross-functional teams to meet data requirements and deliver robust solutions.
• Estimate development efforts and contribute to sprint planning and delivery.
• Participate in code reviews and mentor junior team members.
• Design cost-optimized architectures leveraging GCP-native services.

Skills

Java

Spark

Hadoop

Hive

SQL

Linux

Data pipelines

Airflow

Cloud platforms

Tools

Postman

Shell scripting

Git

Airflow

Cloud Composer

Job description

Gurgaon, Bengaluru

Job Summary

We are looking for skilled GCP Data Engineers with 6–9 years of hands‑on experience in building scalable, high‑performance data solutions.

The ideal candidate will have strong expertise in Java‑based big data processing frameworks and deep exposure to Google Cloud Platform (GCP) services for modern data engineering workloads.

Key Skills & Experience

• Strong programming expertise in Java, with experience in building distributed data processing applications
• Hands‑on experience with Big Data technologies such as Apache Spark (Java/Scala APIs), Hadoop, and Hive
• Experience with Spark DataFrame/Spark SQL using Java or Scala (PySpark knowledge is a plus but not primary)
• Solid understanding of data structures, algorithms, and object‑oriented programming in Java
• Strong knowledge of SQL, data modeling, and data warehousing concepts
• Experience working with Linux/Unix environments and scripting (Bash or similar)
• Proven analytical and problem‑solving skills, especially in debugging and optimizing data pipelines
• Ability to design and build scalable, fault‑tolerant data processing systems

Important to Have

• Hands‑on experience with GCP services such as BigQuery, Dataflow (Apache Beam with Java), Dataproc, Cloud Storage, Pub/Sub, and IAM
• Experience with workflow orchestration tools like Airflow or Cloud Composer
• Exposure to cloud migration projects, especially transitioning from on‑premise Hadoop ecosystems to GCP
• Familiarity with streaming data pipelines using Pub/Sub and Dataflow
• Understanding of CI/CD pipelines and DevOps practices in a cloud environment

Roles & Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Java‑based big data frameworks on GCP
• Build and optimize batch and streaming data processing solutions using Dataproc, Dataflow, and Spark
• Ensure high‑quality, efficient, and maintainable code by following best practices and coding standards
• Perform unit testing, integration testing, and troubleshoot complex data pipeline issues
• Collaborate with cross‑functional teams to understand data requirements and deliver robust solutions
• Estimate development efforts and contribute to sprint planning and delivery
• Participate in code reviews and mentor junior team members where required
• Design cost‑optimized and performance‑efficient architectures leveraging GCP‑native services

Core Java

• Strong knowledge of OOP concepts, Data Structures & Algorithms
• Expertise in Collections, Generics, Exception Handling
• Experience in Multithreading & Basic Concurrency
• Hands‑on with Java IO & file processing
• Understanding of Regular Expressions

Java & Spring Framework

• Experience in Spring Core, Spring Boot
• Strong exposure to REST API design and development
• Knowledge of Microservices architecture

Big Data Technologies

• Hands‑on experience with Hadoop ecosystem, MapReduce, Apache Spark (Core & basics of Spark SQL/PySpark), Hive (good to have), HBase or other NoSQL systems
• Understanding of distributed computing concepts, batch & real‑time data processing pipelines
• Ability to handle large‑scale data (GBs to TBs)

GCP (Google Cloud Platform)

• Hands‑on experience with BigQuery, Cloud Storage (GCS)
• Good to have: Dataflow, Dataproc, Pub/Sub
• Understanding of cloud‑based data pipelines and deployment

Database / Tools

• Strong knowledge of SQL (MySQL / Hive / BigQuery queries) and JSON handling & API integration
• Tools: Postman (API testing), Shell scripting (basic automation), Version control tools (Git preferred)