Big Data Developer

IntraEdge; Chennai District; On-site; INR 1,500,000 - 2,500,000; Full time

Job summary

IntraEdge in Chennai is seeking a Data Engineer with 4–8 years of experience to design, build, and optimize scalable data pipelines on GCP and Hadoop ecosystem. The role focuses on delivering analytics-ready data architectures to empower BI and ML initiatives.

You will work with Spark/Scala, Hive SQL, BigQuery, and Cloud Storage, develop ETL processes, ensure data quality and governance, and collaborate with data analysts and scientists to drive insights.

Qualifications

• Strong experience in Big Data technologies (Hadoop ecosystem preferred).
• Hands-on experience with Scala and Apache Spark.
• Proficiency in Hive SQL and advanced query optimization.
• Experience working with GCP services such as BigQuery and Google Cloud Storage.
• Solid understanding of ETL processes and data pipeline development.
• Strong knowledge of Data Modeling (dimensional and relational models).
• Good understanding of distributed data processing and performance tuning.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Apache Spark and Scala.
• Work with large datasets using Hive SQL and optimize query performance.
• Develop and manage data solutions on Google Cloud Platform (GCP) including Google Cloud Storage and BigQuery.
• Build robust ETL pipelines to ingest, transform, and load data from multiple sources.
• Implement and maintain efficient data models to support analytics and reporting.
• Ensure data quality, integrity, and governance across data platforms.
• Collaborate with cross-functional teams including Data Analysts, Data Scientists, and Business stakeholders.
• Troubleshoot and optimize data workflows for performance and scalability.

Skills

Big Data

Scala

Apache Spark

Hive SQL

GCP

BigQuery

ETL

Data modelling

Distributed processing

Tools

Airflow

Python

Job description

Location: Chennai [Hybrid to client location]

Experience: 4–8 Years

Employment Type: Full-Time

Job Summary

We are looking for a skilled Data Engineer with strong expertise in Big Data technologies and Google Cloud Platform (GCP). The ideal candidate will be responsible for designing, building, and optimizing scalable data pipelines and data architectures to support advanced analytics and business intelligence needs.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Apache Spark and Scala.
• Work with large datasets using Hive SQL and optimize query performance.
• Develop and manage data solutions on Google Cloud Platform (GCP) including Google Cloud Storage and BigQuery.
• Build robust ETL pipelines to ingest, transform, and load data from multiple sources.
• Implement and maintain efficient data models to support analytics and reporting.
• Ensure data quality, integrity, and governance across data platforms.
• Collaborate with cross-functional teams including Data Analysts, Data Scientists, and Business stakeholders.
• Troubleshoot and optimize data workflows for performance and scalability.

Required Skills & Qualifications

• Strong experience in Big Data technologies (Hadoop ecosystem preferred).
• Hands-on experience with Scala and Apache Spark.
• Proficiency in Hive SQL and advanced query optimization.
• Experience working with GCP services such as BigQuery and Google Cloud Storage.
• Solid understanding of ETL processes and data pipeline development.
• Strong knowledge of Data Modeling (dimensional and relational models).
• Good understanding of distributed data processing and performance tuning.

Preferred Qualifications

• Experience with workflow orchestration tools like Airflow.
• Knowledge of Python for data processing.
• Familiarity with data warehousing concepts and modern data architectures.
• Experience in handling real-time or streaming data pipelines is a plus.