Big Data Developer

Daten; Dadri; On-site; INR 1,500,000 - 3,000,000; Full time

Job summary

A data solutions company in India is seeking a skilled Big Data Engineer to design and maintain scalable data solutions. The ideal candidate has over 5 years of experience in data engineering, proficiency with Apache Spark, and expertise in AWS services. Key responsibilities include building efficient data pipelines and developing data processing solutions. This role offers the opportunity to work within a collaborative team in a dynamic tech stack.

Qualifications

• 5+ years of experience in Big Data Engineering or Data Analysis.
• Strong hands-on expertise with Apache Spark.
• Experience working with AWS cloud services.

Responsibilities

• Design, develop, and maintain scalable big data solutions.
• Build and optimize data pipelines for ingestion and storage.
• Develop streaming and batch processing solutions using Spark.

Skills

Apache Spark

SQL expertise

AWS services

Python programming

Hadoop ecosystems

Elasticsearch/OpenSearch

Education

Bachelor’s degree in a related discipline

Tools

Apache Spark

AWS EMR

Hadoop

Kibana

Job description

We are looking for a skilled and passionate Big Data Engineer with a strong background in data analysis and distributed computing. This role is ideal for professionals who enjoy working on high-volume data pipelines, developing scalable data solutions, and leveraging cloud-native services to drive business insights. You will work with a collaborative team of engineers and analysts to build and optimize data systems across a modern tech stack.

Education Requirements

• Bachelor’s degree in a related discipline.

Experience Requirements

• 5+ years of proven experience in Big Data Engineering or Data Analysis roles.
• Strong hands-on expertise with Apache Spark (Core Spark, Spark Streaming, DataFrames, RDD, Spark SQL) working on large-scale data processing.
• Solid experience in writing complex SQL queries and performing SQL performance tuning using Hive or Impala.
• Experience working with AWS cloud services, particularly EMR, Glue, S3, Athena, Lambda, CloudWatch, and IAM in a serverless architecture.
• Proficiency in Python for data transformation, automation, and scripting tasks.
• Working knowledge of Hadoop ecosystems and distributed data processing frameworks.
• Experience with Elasticsearch/OpenSearch and building dashboards using Kibana.
• Familiarity with job schedulers like Airflow, Autosys, or AWS Data Pipeline.
• Experience with Key/Value data stores like HBase.
• Prior exposure to cloud-based infrastructure, data security, and performance optimization is highly desirable.

Role & Responsibilities

• Design, develop, and maintain scalable big data solutions using Apache Spark, Python, and SQL.
• Build and optimize data pipelines and architectures for efficient data ingestion, transformation, and storage.
• Work extensively with AWS services including EMR, Glue, S3, Athena, Lambda, and others to support data processing in a serverless/cloud environment.
• Develop streaming and batch data processing solutions using Spark Streaming and related technologies.