Big Data - Senior Engineer

JobCubby; Hinoba-an; On-site; PHP 796,000 - 1,194,000; Full time

Job summary

Iris Software is seeking a Senior Big Data Engineer to join our on-site teams across Noida, Gurugram, and Pune. You will design scalable data solutions using Spark, Hadoop, and Databricks, and mentor peers in best practices for lakehouse architectures.

The role requires 7-8 years in data engineering, strong SQL/Hive skills, and experience delivering end-to-end data platform projects in a fast-paced environment.

Qualifications

• 7-8 years of data engineering experience with big data ecosystems.
• Experience designing scalable data pipelines and lakehouse architectures.
• Proficient in Spark, Hadoop, Databricks, and SQL optimization.
• Experience mentoring teams and collaborating with stakeholders.

Responsibilities

• Design scalable Big Data solutions using Apache Spark, Hadoop, Azure Databricks, and modern data platform technologies.
• Define data processing architecture, transformation strategies, and engineering standards aligned with business objectives.
• Lead development of distributed data processing pipelines using Spark (Scala or PySpark) and Hadoop ecosystem technologies.
• Design and optimize SQL and Hive-based data processing solutions to improve performance and scalability.
• Architect and optimize Azure Databricks solutions supporting large-scale data engineering and analytics workloads.
• Design and implement data Lakehouse solutions leveraging Apache Hudi or Apache Iceberg.
• Establish data ingestion, transformation, validation, and reconciliation frameworks to improve data reliability.
• Drive performance tuning initiatives across Spark jobs, Databricks workloads, Hive queries, and Hadoop processing environments.
• Review data engineering solutions to ensure adherence to architecture, performance, and engineering standards.
• Troubleshoot complex data processing, performance, and platform issues through detailed root cause analysis.
• Mentor team members on Spark, Hadoop, Databricks, Hudi/Iceberg, SQL optimization, and Big Data engineering best practices.
• Collaborate with various teams and stakeholders to support end-to-end data platform delivery.
• Drive continuous improvement initiatives focused on scalability, performance, reliability, and operational efficiency.

Skills

PySpark

Scala

Spark

SQL

Data modeling

Tools

Sqoop

Hadoop (HDFS + YARN)

Azure Databricks

HBase

Impala

Hive

Apache Hudi

Scala

Job description

Why Join Iris?

Are you ready to do the best work of your career at one of India’s Top 25 Best Workplaces in IT industry? Do you want to grow in an award-winning culture that truly values your talent and ambitions?

Join Iris Software — one of the fastest-growing IT services companies — where you own and shape your success story.

About Us

At Iris Software, our vision is to be our client’s most trusted technology partner, and the first choice for the industry’s top professionals to realize their full potential.

With over 4,300 associates across India, U.S.A, and Canada, we help our enterprise clients thrive with technology-enabled transformation across financial services, healthcare, transportation & logistics, and professional services.

Our work covers complex, mission-critical applications with the latest technologies, such as high-value complex Application & Product Engineering, Data & Analytics, Cloud, DevOps, Data & MLOps, Quality Engineering, and Business Automation.

Working with Us

At Iris, every role is more than a job — it’s a launchpad for growth.

Our Employee Value Proposition, “Build Your Future. Own Your Journey.” reflects our belief that people thrive when they have ownership of their career and the right opportunities to shape it.

We foster a culture where your potential is valued, your voice matters, and your work creates real impact. With cutting-edge projects, personalized career development, continuous learning and mentorship, we support you to grow and become your best — both personally and professionally.

Curious what it’s like to work at Iris?

Head to this video for an inside look at the people, the passion, and the possibilities. Watch it here.

Job Description

Experience: 7-8 Years

Location: Noida, Gurugram, Pune

Mandatory Skills

Data Ingestion Tools (Sqoop), Hadoop (HDFS + YARN), Azure Databricks, Hadoop Ecosystem Fundamentals (HBase + Impala), Hive, Scala, Apache Hudi

Additional Skills

PySpark

Key Responsibilities

• Design scalable Big Data solutions using Apache Spark, Hadoop, Azure Databricks, and modern data platform technologies.
• Define data processing architecture, transformation strategies, and engineering standards aligned with business objectives.
• Lead development of distributed data processing pipelines using Spark (Scala or PySpark) and Hadoop ecosystem technologies.
• Design and optimize SQL and Hive-based data processing solutions to improve performance and scalability.
• Architect and optimize Azure Databricks solutions supporting large-scale data engineering and analytics workloads.
• Design and implement data Lakehouse solutions leveraging Apache Hudi or Apache Iceberg.
• Establish data ingestion, transformation, validation, and reconciliation frameworks to improve data reliability.
• Drive performance tuning initiatives across Spark jobs, Databricks workloads, Hive queries, and Hadoop processing environments.
• Review data engineering solutions to ensure adherence to architecture, performance, and engineering standards.
• Troubleshoot complex data processing, performance, and platform issues through detailed root cause analysis.
• Mentor team members on Spark, Hadoop, Databricks, Hudi/Iceberg, SQL optimization, and Big Data engineering best practices.
• Collaborate with various teams and stakeholders to support end-to-end data platform delivery.
• Drive continuous improvement initiatives focused on scalability, performance, reliability, and operational efficiency.

Behavioral Competencies

• Demonstrates strong ownership while driving Big Data Engineering excellence.
• Collaborate effectively with various teams and business stakeholders to ensure smooth delivery.
• Promotes quality-focused engineering through proactive validation, optimization, and continuous improvement.
• Applies strong analytical thinking to evaluate complex data engineering and platform challenges.
• Demonstrate adaptability while managing evolving technologies, data ecosystems, and business requirements.
• Communicates effectively regarding delivery status, risks, dependencies, and improvement opportunities.
• Maintains high attention to detail across data architecture, processing design, testing, and implementation activities.
• Encourages continuous improvement in data engineering practices and platform operations.
• Supports knowledge sharing and mentoring to strengthen team capabilities.
• Balances scalability, performance, reliability, and business priorities while driving delivery excellence.

Mandatory Competencies

• Programming Language - Scala - Scala
• Big Data - Big Data - HIVE
• Big Data - Big Data - Cloudera
• Cloud - Cloud - Apache Hudi
• Big Data - Big Data - Hadoop
• Big Data - Big Data - Sqoop
• Big Data - Big Data - Pyspark

Perks and Benefits for Irisians

Iris provides world-class benefits for a personalized employee experience. These benefits are designed to support financial, health and well-being needs of Irisians for a holistic professional and personal growth. Click here to view the benefits.