Senior Big Data Engineer

dunnhumby; Gurgaon; On-site; INR 2,800,000 - 4,000,000; Full time

Benefits offered by this job

Flexible working hours

Birthday off

Diversity & Inclusion programs

Job summary

dunnhumby is seeking a Sr Big Data Engineer in Gurgaon to extend and improve our data engineering capabilities and contribute to global data platforms. You will design end-to-end data solutions and collaborate with cross‑functional teams to deliver scalable pipelines.

You should have 5+ years in data architecture or engineering, strong Python/Java/Scala skills, experience with Spark, Kafka and Airflow, and cloud familiarity (Azure or Google Cloud).

Qualifications

• Bachelor's or Master's in Computer Science, Information Systems, Data Science, or related field.
• 5+ years of experience in data architecture/engineering or a related field.
• Proficient in Python, Java or Scala and modern data stack tools.

Responsibilities

• Design end-to-end data solutions, including data lakes, data warehouses, ETL/ELT pipelines, APIs, and analytics platforms.
• Architect scalable and low-latency data pipelines using Spark, Kafka, and similar tools.
• Design and orchestrate automation with Airflow to manage complex workflows and dependencies.
• Detect anomalies, trigger alerts, and reroute or restart processes to maintain data integrity.
• Define and implement data governance, metadata management, and data quality standards.
• Lead architectural reviews and technical design sessions to guide solution development.
• Partner with business and IT teams to translate needs into data architecture requirements.
• Ensure security, compliance, and regulatory requirements are addressed in data solutions.
• Evaluate and suggest improvements to existing data architecture and processes.

Skills

Big Data

Python

Java/Scala

Spark

Kafka

Airflow

Cloud platforms

Gurgaon

Education

Bachelor's or Master's in CS/IS/Data Science

Tools

Spark

Kafka

Airflow

dbt

Snowflake

Databricks

Hadoop/Hive

Job description

dunnhumby is the global leader in Customer Data Science, partnering with the world’s most ambitious retailers and brands to put the customer at the heart of every decision. We combine deep insight, advanced technology, and close collaboration to help our clients grow, innovate, and deliver measurable value for their customers.

dunnhumby employs nearly 2,500 experts in offices throughout Europe, Asia, Africa, and the Americas working for transformative, iconic brands such as Tesco, Coca-Cola, Nestlé, Unilever and Metro.

We’re looking for a Sr Big Data Engineer who expects more from their career. It’s chance to extend and improve dunnhumby’s Data Engineering Team. It’s an opportunity to work with a market-leading business to explore new opportunities for us and influence global retailers.

Key Responsibilities

• Design end-to-end data solutions, including data lakes, data warehouses, ETL/ELT pipelines, APIs, and analytics platforms.
• Architect scalable and low-latency data pipelines using tools like Apache Kafka, Flink, or Spark Streaming to handle high-velocity data streams.
• Design /Orchestrate end-to-end automation using orchestration frameworks such as Apache Airflow to manage complex workflows and dependencies.
• Design intelligent systems that can detect anomalies, trigger alerts, and automatically reroute or restart processes to maintain data integrity and availability.
• Define and implement data governance, metadata management, and data quality standards.
• Lead architectural reviews and technical design sessions to guide solution development.
• Partner with business and IT teams to translate business needs into data architecture requirements.
• Ensure security, compliance, and regulatory requirements are addressed in all data solutions.
• Evaluate and recommend improvements to existing data architecture and processes.

Technical Expertise

• Bachelors or masters degree in computer science, Information Systems, Data Science, or related field.
• Extensive experience with high level programming languages - Python, Java or Scala
• 5+ years of experience in data architecture, data engineering, or a related field.
• Proficient in data pipeline tools such as Apache Spark, Kafka, Airflow, or similar.
• Experience with data governance frameworks and tools (e.g., Collibra, Alation, OpenMetadata).
• Strong knowledge of cloud platforms (Azure or Google Cloud), especially with cloud-native data services.
• Experience working in Agile or DevOps environments.
• Experience with modern data stack tools (e.g., dbt, Snowflake, Databricks).
• Experience with Hive, Oozie, Airflow, HBase, MapReduce, Spark along with working knowledge of Hadoop/Spark Toolsets.
• Extensive Experience working with Git and Process Automation
• In depth understanding of relational database management systems (RDBMS) and Data Flow Development.

What you can expect from us

We won’t just meet your expectations. We’ll defy them. So you’ll enjoy the comprehensive rewards package you’d expect from a leading technology company. But also, a degree of personal flexibility you might not expect. Plus, thoughtful perks, like flexible working hours and your birthday off.

You’ll also benefit from an investment in cutting-edge technology that reflects our global ambition. But with a nimble, small-business feel that gives you the freedom to play, experiment and learn.

And we don’t just talk about diversity and inclusion. We live it every day – with thriving networks including dh Gender Equality Network, dh Proud, dh Family, dh One, dh Enabled and dh Thrive as the living proof. We want everyone to have the opportunity to shine and perform at your best throughout our recruitment process. Please let us know how we can make this process work best for you.

Our approach to Flexible Working

At dunnhumby, we value and respect difference and are committed to building an inclusive culture by creating an environment where you can balance a successful career with your commitments and interests outside of work.

We believe that you will do your best at work if you have a work / life balance. Some roles lend themselves to flexible options more than others, so if this is important to you please raise this with your recruiter, as we are open to discussing agile working opportunities during the hiring process.

For further information about how we collect and use your personal information please see our Privacy Notice which can be found (here)

Global Diversity and Inclusion Statements

At dunnhumby, we utilise our diversity of thought as our competitive edge.

We are proud of our diversity and committed to making dunnhumby an even more inclusive place to work that we can be proud of.

Our diversity and inclusion work is designed to cultivate a culture of belonging, where every dunnhumbian feels safe to bring their whole self to work, where everyone is welcome and we practice what we preach.

We have a full D&I strategy to implement this long-term behaviour change; in addition, we have five employee-led network groups to support colleagues in the areas of gender, sexual orientation, multiculturalism, mental health and wellbeing, and family.