Lead Software Engineer - Data

EPAM Systems; India; On-site; INR 2,800,000 - 7,000,000; Full time

Job summary

EPAM Systems in India seeks a senior Big Data architect to lead design and implementation of enterprise data platforms, delivering scalable pipelines with Spark and Hadoop.

The role emphasizes cloud-native data services on AWS/Azure with Databricks, mentoring engineers, and promoting AGILE practices across the SDLC.

Qualifications

• 8-14 years of experience in Big Data and related data technologies.
• Expertise in distributed computing principles and Apache Spark.
• Proficiency in Hadoop v2, MapReduce, HDFS, and Sqoop.
• Experience with messaging systems such as Kafka or RabbitMQ.
• Familiarity with Big Data querying tools such as Hive and Impala.
• Understanding of SQL queries, joins, stored procedures, and relational schemas.
• Experience with NoSQL databases such as HBase, Cassandra, or MongoDB.
• Knowledge of ETL techniques and frameworks.
• Experience with native cloud data services on Azure or AWS with Databricks.
• Capability to lead a team efficiently.
• Background in designing and implementing Big Data solutions.
• Practitioner of AGILE methodology.

Responsibilities

• Lead the design and implementation of Big Data solutions across the organization.
• Develop and optimize distributed data processing pipelines using Apache Spark.
• Write efficient, scalable code in Python to support data engineering initiatives.
• Build stream-processing systems leveraging technologies such as Apache Storm or Spark-Streaming.
• Integrate data from multiple sources, including RDBMS, ERP systems, and file-based inputs.
• Apply ETL techniques and frameworks to ensure reliable data transformation and delivery.
• Tune and optimize Spark job performance to meet business and technical requirements.
• Guide the team in adopting native cloud data services on Azure or AWS with Databricks.
• Manage and mentor a team of engineers to deliver high-quality data solutions efficiently.
• Drive Agile practices throughout the software development lifecycle.

Skills

Big Data

Distributed Computing

Apache Spark

Hadoop

MapReduce

HDFS

Sqoop

Kafka

RabbitMQ

Hive

Impala

HBase

Cassandra

MongoDB

SQL

ETL

Azure/AWS

Databricks

AGILE

Leadership

Tools

Databricks

Job description

Responsibilities

• Lead the design and implementation of Big Data solutions across the organization Develop and optimize distributed data processing pipelines using Apache Spark Write efficient, scalable code in Python to support data engineering initiatives Build stream-processing systems leveraging technologies such as Apache Storm or Spark-Streaming Integrate data from multiple sources, including RDBMS, ERP systems, and file-based inputs Apply ETL techniques and frameworks to ensure reliable data transformation and delivery Tune and optimize Spark job performance to meet business and technical requirements Guide the team in adopting native cloud data services on Azure or AWS with Databricks Manage and mentor a team of engineers to deliver high-quality data solutions efficiently Drive Agile practices throughout the software development lifecycle

Requirements

• 8-14 years of experience in Big Data and related data technologies Expertise in distributed computing principles and Apache Spark Proficiency in Hadoop v2, MapReduce, HDFS, and Sqoop Experience with messaging systems such as Kafka or RabbitMQ Familiarity with Big Data querying tools such as Hive and Impala Understanding of SQL queries, joins, stored procedures, and relational schemas Experience with NoSQL databases such as HBase, Cassandra, or MongoDB Knowledge of ETL techniques and frameworks Experience with native cloud data services on Azure or AWS with Databricks Capability to lead a team efficiently Background in designing and implementing Big Data solutions Practitioner of AGILE methodology