Big Data Developer

Agile Ventures; Bengaluru; On-site; INR 3,000,000 - 4,200,000; Full time

Job summary

Agile Ventures in Bengaluru seeks a seasoned Big Data Engineer to lead data solutions and drive scalable pipelines. You will work hands-on with Spark, Hadoop, and streaming tech to deliver robust data products.

The role covers 7+ years in data tech, strong SQL/NoSQL skills, and mentoring a team under AGILE practices. Cloud data services experience is a plus. Join a dynamic data-driven team today.

Qualifications

• 7+ years of experience in Big Data and data technologies.
• Expert understanding of distributed computing principles.
• Expert knowledge of Apache Spark and big data tools.
• Hands-on programming with Python.
• Experience with Hadoop v2, MapReduce, HDFS, Sqoop.
• Experience with streaming systems (Storm or Spark Streaming).
• Experience with messaging systems (Kafka or RabbitMQ).
• Strong SQL, joins, stored procedures, relational schemas.
• Experience with NoSQL databases (HBase, Cassandra, MongoDB).
• Knowledge of ETL techniques and performance tuning.
• AGILE practitioner and able to lead a team.

Responsibilities

• Design and implement big data solutions and architectures.
• Lead a team efficiently and collaborate across projects.
• Develop and optimize Spark jobs and data pipelines.

Skills

Big Data

Distributed Computing

Python

SQL

NoSQL

Spark

Hadoop

Stream Processing

Kafka / RabbitMQ

AGILE

Team Leadership

Education

Bachelor's degree in Computer Science or related

Tools

Hadoop v2

Spark

Hive

Impala

Sqoop

Apache Storm / Spark Streaming

Kafka / RabbitMQ

Azure Databricks

Job description

Job Description

1. 7+ Years of in Big Data & Data related technology experience
2. Expert level understanding of distributed computing principles
3. Expert level knowledge and experience in Apache Spark
4. Hands on programming with Python
5. Proficiency with Hadoop v2, Map Reduce, HDFS, Sqoop
6. Experience with building stream-processing systems, using technologies such as Apache Storm or Spark-Streaming
7. Experience with messaging systems, such as Kafka or RabbitMQ
8. Good understanding of Big Data querying tools, such as Hive, and Impala
9. Experience with integration of data from multiple data sources such as RDBMS (SQL Server, Oracle), ERP, Files
10. Good understanding of SQL queries, joins, stored procedures, relational schemas
11. Experience with NoSQL databases, such as HBase, Cassandra, MongoDB
12. Knowledge of ETL techniques and frameworks
13. Performance tuning of Spark Jobs
14. Experience with native Cloud data services Azure databricks
15. Ability to lead a team efficiently
16. Experience with designing and implementing Big data solutions
17. Practitioner of AGILE methodology