Big Data Developer

Statusneo Technology Consulting; Bengaluru; On-site; INR 900,000 - 1,400,000; Full time

Job summary

Statusneo Technology Consulting is seeking a Big Data Developer to design, develop, and maintain large-scale data processing systems and pipelines in Bengaluru. You will work with Spark, Hadoop, Kafka, and Airflow to build scalable, reliable data solutions supporting analytics and ML initiatives.

The role requires strong Python/Scala/Java skills, SQL, and experience with cloud platforms. You'll collaborate across teams and implement data governance and CI/CD practices.

Qualifications

• Bachelor's or Master's degree in CS/IT/Engineering or equivalent.
• ,
• ,
• ,
• ,
• ,
• ,
• ],
• job_responsibilities_description[

Responsibilities

• Design, develop, and maintain scalable big data solutions and pipelines.
• Build and optimize ETL/ELT processes for processing structured and unstructured data.
• Develop data ingestion frameworks from multiple data sources including databases, APIs, streaming systems, and file-based systems.
• Work with distributed data processing technologies such as Spark, Hadoop, and Kafka.
• Develop batch and real-time data processing applications.
• Ensure data quality, integrity, security, and governance across data platforms.
• Monitor and troubleshoot production data workflows and resolve performance issues.
• Collaborate with Data Architects and Business Stakeholders to understand data requirements.
• Implement best practices for data modeling, code quality, testing, and deployment.
• Participate in code reviews and contribute to continuous improvement initiatives.
• Support cloud-based data solutions and migration projects.

Skills

Python

Scala

Java

SQL

Big Data concepts

Data modeling

Agile/Scrum

Education

Bachelor's or Master's degree in CS/IT/Engineering

Tools

Apache Spark

Hadoop

Kafka

Airflow

Databricks

Azure

AWS

GCP

SQL Server

Oracle

PostgreSQL

MongoDB

Cassandra

HBase

Git

Azure DevOps

Jenkins

Job description

Big Data Developer

We are seeking a highly skilled Big Data Developer to design, develop, and maintain large‑scale data processing systems and data pipelines. The ideal candidate will have experience working with distributed computing frameworks, cloud platforms, and modern big data technologies to support data analytics, reporting, and machine learning initiatives.

The candidate will collaborate with data engineers, data scientists, business analysts, and technology teams to build scalable, reliable, and high-performance data solutions.

Key Responsibilities

• Design, develop, and maintain scalable big data solutions and data pipelines.
• Build and optimize ETL/ELT processes for processing structured and unstructured data.
• Develop data ingestion frameworks from multiple data sources including databases, APIs, streaming systems, and file-based systems.
• Work with distributed data processing technologies such as Spark, Hadoop, and Kafka.
• Develop batch and real‑time data processing applications.
• Ensure data quality, integrity, security, and governance across data platforms.
• Monitor and troubleshoot production data workflows and resolve performance issues.
• Collaborate with Data Architects and Business Stakeholders to understand data requirements.
• Implement best practices for data modeling, code quality, testing, and deployment.
• Participate in code reviews and contribute to continuous improvement initiatives.
• Support cloud‑based data solutions and migration projects.

Required Technical Skills

Big Data Technologies

• Apache Spark (PySpark/Scala Spark)
• Hadoop Ecosystem (HDFS, Hive, Yarn)
• Apache Kafka
• Apache Airflow
• Databricks

Programming Languages

• Python
• Scala
• Java
• SQL

Database Technologies

• SQL Server
• Oracle
• PostgreSQL
• MySQL
• NoSQL Databases (MongoDB, Cassandra, HBase)

Cloud Platforms

• Microsoft Azure (ADLS, Azure Data Factory, Synapse Analytics)
• AWS (S3, EMR, Glue, Redshift)
• Google Cloud Platform (BigQuery, DataProc) – Preferred

DevOps & Version Control

• Git
• Azure DevOps
• Jenkins
• CI/CD Pipelines

Required Qualifications

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.
• 3+ years of experience in Big Data development and data engineering projects.
• Strong experience with Spark, Hadoop, Kafka, and distributed computing environments.
• Proficiency in Python, Scala, or Java.
• Strong SQL and data modeling skills.
• Experience working with cloud-based data platforms (Azure/AWS/GCP).
• Understanding of data warehousing concepts and ETL design patterns.
• Experience in Agile/Scrum development methodologies.