Senior Data Engineer

Agile Ventures; Bengaluru; Hybrid; INR 3,500,000 - 5,500,000; Full time

Job summary

Agile Ventures is seeking an experienced Big Data Engineer to join our team in Bengaluru. You will work on large-scale data processing systems, leveraging Apache Spark, Hadoop ecosystem, and cloud services to deliver robust data solutions.

The role requires 8+ years of experience, strong Python programming skills, and expertise in streaming, SQL, NoSQL, and data integration across sources. You will also lead a small team and contribute to agile development practices.

Qualifications

• 8+ years in big data and data technologies.
• Expert in distributed computing principles.
• Expert in Apache Spark.
• Hands-on Python programming.
• Experience with Hadoop v2, MapReduce, HDFS, Sqoop.
• Experience with streaming systems like Storm or Spark Streaming.
• Experience with messaging systems such as Kafka or RabbitMQ.
• Strong SQL, joins, stored procedures, relational schemas.
• Experience with NoSQL databases such as HBase, Cassandra, MongoDB.
• Knowledge of ETL techniques and frameworks.
• Performance tuning of Spark jobs.
• Experience with Google Cloud Platform.
• Ability to lead a team efficiently.
• Experience designing and implementing Big Data solutions.
• Practitioner of AGILE methodology.

Skills

Big data concepts

Distributed computing

Apache Spark

Python programming

Hadoop ecosystem

SQL querying

NoSQL databases

Streaming systems

Data integration

AGILE methodology

Tools

Hadoop

MapReduce

HDFS

Sqoop

Storm

Spark Streaming

Kafka

RabbitMQ

Hive

Impala

MongoDB

Cassandra

Job description

Job Description:

1. 8+ Years of in Big Data & Data related technology experience
2. Expert level understanding of distributed computing principles
3. Expert level knowledge and experience in Apache Spark
4. Hands on programming with Python
5. Proficiency with Hadoop v2, Map Reduce, HDFS, Sqoop
6. Experience with building stream-processing systems, using technologies such as Apache Storm or Spark-Streaming
7. Experience with messaging systems, such as Kafka or RabbitMQ
8. Good understanding of Big Data querying tools, such as Hive, and Impala
9. Experience with integration of data from multiple data sources such as RDBMS (SQL Server, Oracle), ERP, Files
10. Good understanding of SQL queries, joins, stored procedures, relational schemas
11. Experience with NoSQL databases, such as HBase, Cassandra, MongoDB
12. Knowledge of ETL techniques and frameworks
13. Performance tuning of Spark Jobs
14. Experience with native Cloud data services Google Cloud Platform
15. Ability to lead a team efficiently
16. Experience with designing and implementing Big data solutions
17. Practitioner of AGILE methodology

WE OFFER

1. Opportunity to work on technical challenges that may impact across geographies
2. Vast opportunities for self-development: online university, knowledge sharing opportunities globally, learning opportunities through external certifications
3. Opportunity to share your ideas on international platforms
4. Sponsored Tech Talks & Hackathons
5. Possibility to relocate to any EPAM office for short and long-term projects
6. Focused individual development
7. Benefit package:

• Health benefits, Medical Benefits
• Retirement benefits
• Paid time off
• Flexible benefits