Big Data Engineer

Epam Systems; Bangalore Rural; On-site; INR 1,800,000 - 3,200,000; Full time

Job summary

Epam Systems Bangalore is seeking a senior big data engineer to design and implement data solutions using Spark, Hadoop, and related technologies. You will lead data-driven initiatives, mentor juniors, and ensure robust data pipelines across on-prem and cloud environments.

Ideal candidates have 6-15 years of experience in big data, strong Python programming skills, knowledge of streaming architectures with Kafka, Storm, or Spark Streaming, and a solid grounding in SQL, NoSQL databases, and AGILE

Qualifications

• 6-15 years of experience in Big Data & Data related technology.
• Expert level understanding of distributed computing principles.
• Hands-on programming with Python.
• Experience with Hadoop v2, Map Reduce, HDFS, Sqoop.
• Experience with messaging systems such as Kafka or RabbitMQ.
• Knowledge of ETL techniques and frameworks.

Skills

Big Data

Python

SQL

NoSQL

Hive/Impala

Tools

Apache Spark

Hadoop v2

Kafka

Storm

Spark Streaming

Databricks

Job description

Job Description:

6-15 Years of in Big Data & Data related technology experience

Expert level understanding of distributed computing principles

Expert level knowledge and experience in Apache Spark

Hands on programming with Python

Proficiency with Hadoop v2, Map Reduce, HDFS, Sqoop

Experience with building stream-processing systems, using technologies such as Apache Storm or Spark-Streaming

Experience with messaging systems, such as Kafka or RabbitMQ

Good understanding of Big Data querying tools, such as Hive, and Impala

Experience with integration of data from multiple data sources such as RDBMS (SQL Server, Oracle), ERP, Files

Good understanding of SQL queries, joins, stored procedures, relational schemas

Experience with NoSQL databases, such as HBase, Cassandra, MongoDB

Knowledge of ETL techniques and frameworks

Performance tuning of Spark Jobs

Experience with native Cloud data services AWS Databricks, Azure databricks

Ability to lead a team efficiently

Experience with designing and implementing Big data solutions

Practitioner of AGILE methodology