Big Data Engineer

Yantran; Chennai District; On-site; INR 900,000 - 1,300,000; Full time

Job summary

Yantran in India is seeking a Big Data Engineer to design and develop distributed data-processing platforms capable of handling large-scale datasets.

You will build batch and streaming pipelines, develop Spark/PySpark applications, implement Kafka-based streaming solutions, and optimize data processing across cloud environments.

The role involves building data lakes and analytics platforms with strong data quality and monitoring to support data-driven decision making.

Qualifications

• Experience designing distributed data-processing platforms at scale.
• Proficiency with batch and streaming data pipelines.
• Strong knowledge of Spark, PySpark, Kafka, and SQL.

Responsibilities

• Develop distributed data-processing applications.
• Build batch and streaming pipelines.
• Process high-volume structured and unstructured data.
• Develop Spark/PySpark applications.
• Implement Kafka-based streaming solutions.
• Optimize large-scale data processing.
• Build data lakes and analytics platforms.
• Implement data quality and monitoring.

Skills

Big Data

Apache Spark

PySpark

Hadoop ecosystem

Kafka

Python/Java/Scala

SQL

Databricks

AWS/Azure/GCP

Data Lake technologies

Tools

Databricks

Cloud platforms

Job description

Seeking a Big Data Engineer to design and develop distributed data-processing platforms capable of handling large-scale datasets.

Responsibilities:

• Develop distributed data-processing applications.
• Build batch and streaming pipelines.
• Process high-volume structured and unstructured data.
• Develop Spark/PySpark applications.
• Implement Kafka-based streaming solutions.
• Optimize large-scale data processing.
• Build data lakes and analytics platforms.
• Implement data quality and monitoring.
• Support cloud-based big-data environments.

Required Skills:

• Big Data.
• Apache Spark.
• PySpark.
• Hadoop ecosystem.
• Kafka.
• Python/Java/Scala.
• SQL.
• Databricks preferred.
• AWS/Azure/GCP.
• Data Lake technologies.