Big Data Developer

Innova Solutions; Hyderabad, Chennai District; Hybrid; INR 1,800,000 - 3,200,000; Full time

Job summary

Innova Solutions is seeking a senior data engineer with 7–10 years of hands-on experience in the big data stack. The ideal candidate will work with Apache Spark, Hadoop, Hive and HDFS, and must have Python, PySpark and Java proficiency.

Mandatory exposure to Airflow or Oozie, and containerization with Docker and CI/CD pipelines. Azure cloud experience is preferred, with knowledge of streaming via Kafka, Flink or Storm.

Qualifications

• At least one SQL/NoSQL database experience is mandatory.
• Python, PySpark and Java proficiency required.
• Experience with Airflow or Oozie is mandatory (at least one).
• Docker or GitLab CI/CD experience required (at least one).
• Strong data modelling, governance and security practices mandatory.
• Strong problem-solving and performance tuning skills required.
• Hands-on with cloud platforms (Azure preferred).
• Familiarity with real-time streaming concepts (Kafka/Flink/Storm) is a plus.

Skills

Python

PySpark

Java

Data modeling

Governance

Security

Problem solving

Performance tuning

Cloud platforms

Tools

Spark

Hadoop

Hive

HDFS

Postgres

Cassandra

MongoDB

HBase

Airflow

Oozie

Docker

GitLab CI/CD

Azure

Kafka

Flink

Storm

Job description

Key Responsibilities :

1. Apache Spark, Hadoop, Hive, HDFS -Mandatory & Iceberg(Optional)
2. SQL & NoSQLdatabases (Postgres, Cassandra, MongoDB, HBase) -Mandatory (at least one)
3. Strong programming skills in Python, pySpark(Spark Core) & Java. -Pyspark Mandatory
4. Experience with workflow orchestration tools (Airflow, Oozie).Mandatory(at least one)
5. Proficiency in containerization & CI/CD (Docker, GitLab CI/CD). Mandatory (at least one)
6. Good understanding ofdatamodelling, governance, and security practices. -Mandatory
7. Strong problem-solving and performance tuning skills. -Mandatory
8. Hands-on with cloud platforms (Azure preferred). -Optional
9. Worked on Kafka / Flink / Storm (real-time streaming) -Optional (But should have minimum knowledge on streaming pipeline).

Work Location- Hyd/Chennai

Experience:7 to 10 years

L1/ L2 ( 1 round F2F Mandatory ) followed by clientround.