Senior Big Data Engineer: Spark, Hadoop & Lakehouse

Iris Software, Inc.; Hinoba-an; On-site; PHP 1,990,000 - 3,316,000; Full time

Job summary

Iris Software, Inc. Noida, Gurugram and Pune is looking for a seasoned Big Data Engineer to design scalable data platforms using Spark, Hadoop, and Azure Databricks to support enterprise analytics.

You will architect Lakehouse solutions with Apache Hudi or Iceberg, establish robust ingestion and transformation pipelines, and mentor teammates on best practices in data engineering. This role offers exposure to cutting-edge tech and collaborative projects in a growth-focused environment.

Qualifications

• Experience with data ingestion tools such as Sqoop and Hadoop ecosystems.
• Proficient in Azure Databricks and Hadoop ecosystem fundamentals (HBase, Impala).
• Strong SQL and Hive-based data processing skills for scalable pipelines.
• Experience with Hudi/Iceberg lakehouse architectures and Spark-based workloads.
• PySpark experience is a plus.

Responsibilities

• Design scalable Big Data solutions using Spark, Hadoop, Azure Databricks, and modern data platform technologies.
• Lead development of distributed data processing pipelines using Spark (Scala or PySpark) and Hadoop ecosystem technologies.
• Design and optimize SQL and Hive-based data processing solutions to improve performance and scalability.
• Architect and optimize Azure Databricks solutions supporting large-scale data engineering and analytics workloads.
• Design and implement data Lakehouse solutions leveraging Apache Hudi or Apache Iceberg.
• Establish data ingestion, transformation, validation, and reconciliation frameworks to improve data reliability.
• Drive performance tuning initiatives across Spark jobs, Databricks workloads, Hive queries, and Hadoop processing environments.
• Review data engineering solutions to ensure adherence to architecture, performance, and engineering standards.
• Troubleshoot complex data processing, performance, and platform issues through detailed root cause analysis.
• Mentor team members on Spark, Hadoop, Databricks, Hudi/Iceberg, SQL optimization, and Big Data engineering best practices.
• Collaborate with various teams and stakeholders to support end-to-end data platform delivery.
• Drive continuous improvement initiatives focused on scalability, performance, reliability, and operational efficiency.
• Demonstrates strong ownership while driving Big Data Engineering excellence.
• Collaborate effectively with various teams and business stakeholders to ensure smooth delivery.
• Promotes quality-focused engineering through proactive validation, optimization, and continuous improvement.
• Applies strong analytical thinking to evaluate complex data engineering and platform challenges.
• Demonstrate adaptability while managing evolving technologies, data ecosystems, and business requirements.
• Communicates effectively regarding delivery status, risks, dependencies, and improvement opportunities.
• Maintains high attention to detail across data architecture, processing design, testing, and implementation activities.
• Encourages continuous improvement in data engineering practices and platform operations.
• Supports knowledge sharing and mentoring to strengthen team capabilities.
• Balances scalability, performance, reliability, and business priorities while driving delivery excellence.

Skills

Data Ingestion Tools

Hadoop

Azure Databricks

Hadoop Ecosystem Fundamentals

Hive

Scala

Apache Hudi

PySpark

Job description

Iris Software, Inc. Noida, Gurugram and Pune is looking for a seasoned Big Data Engineer to design scalable data platforms using Spark, Hadoop, and Azure Databricks to support enterprise analytics.

You will architect Lakehouse solutions with Apache Hudi or Iceberg, establish robust ingestion and transformation pipelines, and mentor teammates on best practices in data engineering. This role offers exposure to cutting-edge tech and collaborative projects in a growth-focused environment.