Azure Data Engineer

Epam Systems; Chennai District; Hybrid; INR 1,800,000 - 3,000,000; Full time

Job summary

Epam Systems in Chennai, India seeks a seasoned Big Data Engineer with 6-15 years of experience to lead data engineering efforts using Spark, Hadoop, and related technologies.

You will design and implement scalable data pipelines, work with streaming and batch processing, and collaborate across teams using AGILE methodologies. Strong Python, SQL, NoSQL, and cloud data services knowledge are required.

Qualifications

• 6-15 years of Big Data experience
• Expert level distributed computing principles
• Expert level knowledge and experience in Apache Spark
• Hands on programming with Python
• Proficiency with Hadoop v2, Map Reduce, HDFS, Sqoop
• Experience with building stream-processing systems, using technologies such as Apache Storm or Spark-Streaming
• Experience with messaging systems, such as Kafka or RabbitMQ
• Good understanding of Big Data querying tools, such as Hive, and Impala
• Experience with integration of data from multiple data sources such as RDBMS (SQL Server, Oracle), ERP, Files
• Good understanding of SQL queries, joins, stored procedures, relational schemas
• Experience with NoSQL databases, such as HBase, Cassandra, MongoDB
• Knowledge of ETL techniques and frameworks
• Performance tuning of Spark Jobs
• Experience with native Cloud data services AWS Databricks, Azure databricks
• Ability to lead a team efficiently
• Experience with designing and implementing Big data solutions
• Practitioner of AGILE methodology

Skills

Big Data

Python

Spark

Spark Streaming

Kafka

NoSQL

SQL

ETL

AGILE

Databricks

Hadoop

Hive

Impala

Tools

Hadoop

Spark

Kafka

RabbitMQ

Hive

Impala

HBase

Cassandra

MongoDB

Databricks

SQL Server

Oracle

Sqoop

Storm

Job description

Job Description:

6-15 Years of in Big Data & Data related technology experience

Expert level understanding of distributed computing principles

Expert level knowledge and experience in Apache Spark

Hands on programming with Python

Proficiency with Hadoop v2, Map Reduce, HDFS, Sqoop

Experience with building stream-processing systems, using technologies such as Apache Storm or Spark-Streaming

Experience with messaging systems, such as Kafka or RabbitMQ

Good understanding of Big Data querying tools, such as Hive, and Impala

Experience with integration of data from multiple data sources such as RDBMS (SQL Server, Oracle), ERP, Files

Good understanding of SQL queries, joins, stored procedures, relational schemas

Experience with NoSQL databases, such as HBase, Cassandra, MongoDB

Knowledge of ETL techniques and frameworks

Performance tuning of Spark Jobs

Experience with native Cloud data services AWS Databricks, Azure databricks

Ability to lead a team efficiently

Experience with designing and implementing Big data solutions

Practitioner of AGILE methodology