BigData Engineer

Global Infovision Private Limited; Chennai; On-site; INR 900,000 - 1,500,000; Full time

Job summary

Global Infovision Private Limited is seeking a Data Engineer to design, develop, and maintain scalable data pipelines using Python, PySpark, Spark, and SQL in Chennai. The role involves working with Big Data technologies like Hadoop and Hive to support large-scale data processing and analytics.

You will develop ETL/ELT workflows, optimize queries, and implement data integration solutions. Exposure to Kubernetes, Docker, CI/CD, REST APIs, and GraphQL will be valuable for collaborative deployments

Qualifications

• Strong programming experience in Python or Java or Scala.
• Hands-on experience with Apache Spark / PySpark.
• Strong understanding of Big Data technologies – Hadoop and Hive.
• Good knowledge of SQL and database concepts.
• Experience in building and maintaining data processing pipelines.
• Understanding of REST API integration.
• Exposure to Kubernetes, Docker, and CI/CD practices.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Python, PySpark, and Spark.
• Work with Big Data technologies, including Hadoop and Hive, for large-scale data processing.
• Develop and optimize SQL queries for data transformation and processing.
• Build and maintain ETL/ELT workflows and data integration pipelines.
• Work with REST APIs and GraphQL for data integration and application connectivity.
• Implement and manage data workflows using Apache Airflow.
• Perform data modelling, data warehousing, and data transformation activities.
• Work with distributed computing frameworks and optimize Spark jobs.
• Collaborate with DevOps teams for deployment, containerization, and CI/CD implementation.
• Ensure data quality, reliability, scalability, and performance across data platforms.

Skills

Python

Java

Scala

SQL

REST APIs

GraphQL

Airflow

PySpark

Kubernetes

CI/CD practices

Docker

Spark

Tools

Apache Spark

PySpark

Hadoop

Hive

Job description

We are looking for skilled Data Engineers with strong experience in Python, PySpark, SQL, and Big Data technologies to design, develop, and maintain scalable data processing solutions.

The ideal candidate should have hands-on experience working with distributed data processing frameworks, Hadoop ecosystems, data pipelines, and modern data engineering platforms. Candidates with exposure to Kubernetes, CI/CD, REST APIs, and cloud-native data technologies will be preferred.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Python, PySpark, and Spark.
• Work with Big Data technologies, including Hadoop and Hive, for large-scale data processing.
• Develop and optimize SQL queries for data transformation and processing.
• Build and maintain ETL/ELT workflows and data integration pipelines.
• Work with REST APIs and GraphQL for data integration and application connectivity.
• Implement and manage data workflows using Apache Airflow.
• Perform data modelling, data warehousing, and data transformation activities.
• Work with distributed computing frameworks and optimize Spark jobs.
• Collaborate with DevOps teams for deployment, containerization, and CI/CD implementation.
• Ensure data quality, reliability, scalability, and performance across data platforms.

Mandatory Skills

• Strong programming experience in Python or Java or Scala.
• Hands-on experience with Apache Spark / PySpark.
• Strong understanding of Big Data technologies – Hadoop and Hive.
• Good knowledge of SQL and database concepts.
• Experience in building and maintaining data processing pipelines.
• Understanding of REST API integration.
• Exposure to Kubernetes, Docker, and CI/CD practices.

Good-to-Have Skills

• REST APIs and GraphQL.
• Data Warehousing and Data Modelling.
• Kubernetes and CI/CD implementation.
• Apache Ranger and Trino.
• RDBMS concepts.
• Exposure to SAP FSDM / FPSL.
• Experience with distributed data processing and modern Big Data architectures.