Big Data - Senior Engineer

Iris Software; Dadri; On-site; INR 1,200,000 - 2,000,000; Full time

Job summary

Iris Software is seeking a Senior Big Data Engineer to design scalable data platforms and pipelines across Spark, Hadoop, and Databricks. You will define architecture, transformation strategies, and standards to align with business objectives, leading distributed data processing under a modern data stack.

You'll architect Lakehouse solutions with Hudi or Iceberg, establish robust ingestion and validation frameworks, and mentor the team on performance tuning and best practices across the Hadoop

Qualifications

• 5-8 years of experience in Big Data platforms preferred.
• Proficiency with Spark, Hadoop, and Databricks.
• Strong SQL and data modeling skills.

Responsibilities

• Design scalable Big Data solutions using Spark, Hadoop, Azure Databricks, and modern data platform technologies.
• Define data processing architecture, transformation strategies, and engineering standards aligned with business objectives.
• Lead development of distributed data processing pipelines using Spark (Scala or PySpark) and Hadoop ecosystem technologies.
• Design and optimize SQL and Hive-based data processing solutions to improve performance and scalability.
• Architect and optimize Azure Databricks solutions supporting large-scale data engineering and analytics workloads.
• Design and implement data Lakehouse solutions leveraging Apache Hudi or Apache Iceberg.
• Establish data ingestion, transformation, validation, and reconciliation frameworks to improve data reliability.
• Drive performance tuning initiatives across Spark jobs, Databricks workloads, Hive queries, and Hadoop processing environments.
• Review data engineering solutions to ensure adherence to architecture, performance, and engineering standards.
• Troubleshoot complex data processing, performance, and platform issues through detailed root cause analysis.
• Mentor team members on Spark, Hadoop, Databricks, Hudi/Iceberg, SQL optimization, and Big Data engineering best practices.
• Collaborate with various teams and stakeholders to support end-to-end data platform delivery.
• Drive continuous improvement initiatives focused on scalability, performance, reliability, and operational efficiency.

Skills

Azure Databricks

Data Ingestion Tools

Hadoop (HDFS + YARN)

Hadoop Ecosystem Fundamentals

Scala

Cloudera

SQL

Apache Iceberg

Tools

Sqoop

Job description

Job Summary

Big Data - Senior Engineer

Location

Noida, Gurugram, Pune

Experience

5-8 Years

Mandatory Skills

• Azure Databricks
• Data Ingestion Tools (Sqoop)
• Hadoop (HDFS + YARN)
• Hadoop Ecosystem Fundamentals (HBase + Impala)
• Scala
• Cloudera
• SQL
• Apache Iceberg

Key Responsibilities

• Design scalable Big Data solutions using Apache Spark, Hadoop, Azure Databricks, and modern data platform technologies.
• Define data processing architecture, transformation strategies, and engineering standards aligned with business objectives.
• Lead development of distributed data processing pipelines using Spark (Scala or PySpark) and Hadoop ecosystem technologies.
• Design and optimize SQL and Hive-based data processing solutions to improve performance and scalability.
• Architect and optimize Azure Databricks solutions supporting large-scale data engineering and analytics workloads.
• Design and implement data Lakehouse solutions leveraging Apache Hudi or Apache Iceberg.
• Establish data ingestion, transformation, validation, and reconciliation frameworks to improve data reliability.
• Drive performance tuning initiatives across Spark jobs, Databricks workloads, Hive queries, and Hadoop processing environments.
• Review data engineering solutions to ensure adherence to architecture, performance, and engineering standards.
• Troubleshoot complex data processing, performance, and platform issues through detailed root cause analysis.
• Mentor team members on Spark, Hadoop, Databricks, Hudi/Iceberg, SQL optimization, and Big Data engineering best practices.
• Collaborate with various teams and stakeholders to support end-to-end data platform delivery.
• Drive continuous improvement initiatives focused on scalability, performance, reliability, and operational efficiency.

Behavioral Competencies

• Demonstrates strong ownership while driving Big Data Engineering excellence.
• Collaborate effectively with various teams and business stakeholders to ensure smooth delivery.
• Promotes quality-focused engineering through proactive validation, optimization, and continuous improvement.
• Applies strong analytical thinking to evaluate complex data engineering and platform challenges.
• Demonstrate adaptability while managing evolving technologies, data ecosystems, and business requirements.
• Communicates effectively regarding delivery status, risks, dependencies, and improvement opportunities.
• Maintains high attention to detail across data architecture, processing design, testing, and implementation activities.
• Encourages continuous improvement in data engineering practices and platform operations.
• Supports knowledge sharing and mentoring to strengthen team capabilities.
• Balances scalability, performance, reliability, and business priorities while driving delivery excellence.

Mandatory Competencies

• Big Data - Big Data - Azure Databricks
• Big Data - Big Data - Cloudera
• Programming Language - Scala - Scala
• Big Data - Big Data - Hadoop
• Big Data - Big Data - Impala
• Big Data - Big Data - HDFS
• Big Data - Big Data - Yarn
• Cloud - Cloud - Apache Iceberg
• Database - Query Language - SQL