Big Data Engineer

Epam Systems; Hyderabad, Pune District, Bengaluru; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

Epam Systems is seeking a senior Big Data Engineer in Hyderabad to design and implement scalable data pipelines using Spark and related technologies. You will lead a team, ship streaming and batch processes, and work with data sources such as RDBMS, files, and ERP systems.

The role requires 6-15 years of experience, strong Python coding, and deep knowledge of Hadoop, Spark, and NoSQL databases. AGILE mindset and cloud data services experience are a plus.

Qualifications

• 6-15 years of experience in Big Data & related technology.
• Expert level understanding of distributed computing principles.
• Expert level knowledge and experience in Apache Spark.
• Hands on programming with Python.
• Proficiency with Hadoop v2, Map Reduce, HDFS, Sqoop.
• Experience with building stream-processing systems (e.g., Storm or Spark-Streaming).
• Experience with messaging systems (Kafka or RabbitMQ).
• Strong SQL querying, joins, and stored procedures in relational schemas.
• Experience with NoSQL databases (HBase, Cassandra, MongoDB).
• Knowledge of ETL techniques and frameworks.
• Performance tuning of Spark Jobs.
• Experience with cloud data services (Databricks, GCP).
• Ability to lead a team efficiently.
• Experience with designing and implementing Big Data solutions.
• Practitioner of AGILE methodology.

Responsibilities

• Lead a team to design and implement big data solutions.
• Work with stakeholders to translate requirements into scalable data pipelines.
• Optimize Spark jobs and data workflows for performance and cost.
• Collaborate across data engineering, analytics, and product teams.

Skills

Big Data

Apache Spark

Python

Hadoop

SQL

NoSQL

Kafka

Storm

Hive/Impala

ETL

AGILE

Databricks

GCP

Leadership

Tools

Apache Spark

Hadoop v2

MapReduce

HDFS

Sqoop

Kafka

RabbitMQ

Hive

Impala

NoSQL (HBase/Cassandra/MongoDB)

Job description

Job Description:

6-15 Years of in Big Data & Data related technology experience

Expert level understanding of distributed computing principles

Expert level knowledge and experience in Apache Spark

Hands on programming with Python

Proficiency with Hadoop v2, Map Reduce, HDFS, Sqoop

Experience with building stream-processing systems, using technologies such as Apache Storm or Spark-Streaming

Experience with messaging systems, such as Kafka or RabbitMQ

Good understanding of Big Data querying tools, such as Hive, and Impala

Experience with integration of data from multiple data sources such as RDBMS (SQL Server, Oracle), ERP, Files

Good understanding of SQL queries, joins, stored procedures, relational schemas

Experience with NoSQL databases, such as HBase, Cassandra, MongoDB

Knowledge of ETL techniques and frameworks

Performance tuning of Spark Jobs

Experience with native Cloud data services databricks, GCP

Ability to lead a team efficiently

Experience with designing and implementing Big data solutions

Practitioner of AGILE methodology