Senior Big Data Engineer - Spark, Databricks & Lakehouse

JobCubby; Hinoba-an; On-site; PHP 796,000 - 1,194,000; Full time

Job summary

Iris Software is seeking a Senior Big Data Engineer to join our on-site teams across Noida, Gurugram, and Pune. You will design scalable data solutions using Spark, Hadoop, and Databricks, and mentor peers in best practices for lakehouse architectures.

The role requires 7-8 years in data engineering, strong SQL/Hive skills, and experience delivering end-to-end data platform projects in a fast-paced environment.

Qualifications

• 7-8 years of data engineering experience with big data ecosystems.
• Experience designing scalable data pipelines and lakehouse architectures.
• Proficient in Spark, Hadoop, Databricks, and SQL optimization.
• Experience mentoring teams and collaborating with stakeholders.

Responsibilities

• Design scalable Big Data solutions using Apache Spark, Hadoop, Azure Databricks, and modern data platform technologies.
• Define data processing architecture, transformation strategies, and engineering standards aligned with business objectives.
• Lead development of distributed data processing pipelines using Spark (Scala or PySpark) and Hadoop ecosystem technologies.
• Design and optimize SQL and Hive-based data processing solutions to improve performance and scalability.
• Architect and optimize Azure Databricks solutions supporting large-scale data engineering and analytics workloads.
• Design and implement data Lakehouse solutions leveraging Apache Hudi or Apache Iceberg.
• Establish data ingestion, transformation, validation, and reconciliation frameworks to improve data reliability.
• Drive performance tuning initiatives across Spark jobs, Databricks workloads, Hive queries, and Hadoop processing environments.
• Review data engineering solutions to ensure adherence to architecture, performance, and engineering standards.
• Troubleshoot complex data processing, performance, and platform issues through detailed root cause analysis.
• Mentor team members on Spark, Hadoop, Databricks, Hudi/Iceberg, SQL optimization, and Big Data engineering best practices.
• Collaborate with various teams and stakeholders to support end-to-end data platform delivery.
• Drive continuous improvement initiatives focused on scalability, performance, reliability, and operational efficiency.

Skills

PySpark

Scala

Spark

SQL

Data modeling

Tools

Sqoop

Hadoop (HDFS + YARN)

Azure Databricks

HBase

Impala

Hive

Apache Hudi

Scala

Job description

Iris Software is seeking a Senior Big Data Engineer to join our on-site teams across Noida, Gurugram, and Pune. You will design scalable data solutions using Spark, Hadoop, and Databricks, and mentor peers in best practices for lakehouse architectures.

The role requires 7-8 years in data engineering, strong SQL/Hive skills, and experience delivering end-to-end data platform projects in a fast-paced environment.