Senior Data Engineer

Win Global PR; Bengaluru; On-site; INR 2,800,000 - 6,000,000; Full time

Benefits offered by this job

Positive workplace

Dynamic environment

Inclusive culture

Learning opportunities

Training courses access

Job summary

PayU seeks a Senior Data Engineer to design, implement, and maintain scalable data infrastructure and ML feature pipelines. You will work with Python, SQL, and NoSQL databases, building ETL/ELT workflows on Spark and Airflow across AWS services.

You will contribute to real-time data processing, data modelling, and ML model deployment, enabling data scientists to focus on modelling while you ensure reliability and performance of data platforms.

Qualifications

• Strong expertise in distributed computing and analytical skills.
• Deep understanding of data warehouse design, optimization, and performance tuning.
• Expertise in Python, SQL and NoSQL databases.
• In-depth experience with Cloud services, preferably AWS (EMR, S3, Redshift, MSK).
• Expertise in Spark (PySpark), Airflow, Debezium, Hadoop in data engineering stacks.
• Experience in data modelling and DataMart development.
• Understanding of ML model deployment and feature-store architectures.

Responsibilities

• Partner with business stakeholders to translate data warehouse requirements into technical solutions.
• Support data warehouse initiatives including requirements gathering, design, and implementation.
• Optimize data warehouse performance via query optimization and architectural improvements.
• Design, develop, and maintain robust ETL/ELT pipelines with modern tools.
• Implement real-time and batch data processing to meet business needs.
• Develop and maintain data models and DataMarts for analytics and reporting.
• Deploy and maintain ML model infrastructure and feature pipelines.

Skills

Python

SQL

NoSQL databases

Distributed computing

Data modelling

Tools

Spark (PySpark)

Airflow

Debezium

Hadoop

AWS (EMR, S3, Redshift, MSK)

Job description

Senior Data Engineer

About The Team

Our Data Engineering team is a dynamic and innovative group that forms the backbone of our data-driven organization. We are responsible for building and maintaining robust, scalable data infrastructure that powers critical business decisions across multiple domains. The team works on cutting-edge technologies and handles massive data volumes, ensuring seamless data flow from various sources to our data warehouse and analytics platforms. We play a crucial role in powering Machine Learning models across various use cases by building sophisticated feature engineering pipelines, deploying ML model infrastructure, and maintaining real-time feature generation systems.

About The Role

As a Senior Data Engineer, you will play a crucial role in designing, implementing, and maintaining our data infrastructure. You'll work with modern data engineering technologies and cloud platforms to build scalable ETL pipelines, optimize data processing workflows, and ensure data quality and reliability across our systems. This role extends significantly into the ML domain, where you'll be responsible for building and maintaining feature engineering pipelines that power various ML models across the organization. You'll work on real-time feature generation systems, deploy and maintain ML model infrastructure, and ensure seamless integration between data pipelines and ML workflows. Your work will directly impact model performance and enable data scientists to focus on model development rather than infrastructure concerns.

Responsibilities

• Partner closely with business stakeholders to understand and translate data warehouse requirements into technical solutions
• Provide comprehensive support for data warehouse initiatives including requirements gathering, solution design, and implementation
• Optimize data warehouse performance through query optimization, indexing strategies, and architectural improvements
• Design, develop, and maintain robust ETL/ELT pipelines using modern data engineering tools
• Implement real-time and batch data processing solutions to meet business requirements
• Develop and maintain data models and DataMart solutions for analytics and reporting needs
• Deploy and maintain ML model infrastructure ensuring scalability and reliability

Requirements

• 3-6 years of experience in Data Engineering
• Strong expertise in distributed computing and advanced problem-solving/analytical skills
• Deep understanding of data warehouse design, optimization, and performance tuning
• Expertise in Python, SQL and NoSQL databases
• In-depth experience in Could services, preferably AWS (EMR, S3, Redshift, MSK)
• Expertise in Data engineering tech stack like Spark (PySpark), Airflow, Debezium, Hadoop
• Experience in data modelling, DataMart development
• Understanding of ML model deployment and maintenance workflows with feature store architectures and best practices
• Good to Have - Experience in setting up and working with On-premises Data Engineering tech stack

What we offer?

• A positive, get-things-done workplace
• A dynamic, constantly evolving space (change is par for the course – important you are comfortable with this).
• An inclusive environment that ensures we listen to a diverse range of voices when making decisions.
• Ability to learn cutting edge concepts and innovation in an agile start-up environment with a global scale.
• Access to 5000+ training courses accessible anytime/anywhere to support your growth and development (Corporate with top learning partners like Harvard, Coursera, Udacity).

About Us

At PayU, we are a global fintech investor and our vision is to build a world without financial borders where everyone can prosper. We give people in high growth markets the financial services and products they need to thrive. Our expertise in 18+ high-growth markets enables us to extend the reach of financial services. This drives everything we do, from investing in technology entrepreneurs to offering credit to underserved individuals, to helping merchants buy, sell, and operate online. Being part of Prosus, one of the largest technology investors in the world, gives us the presence and expertise to make a real impact. Find out more at www.payu.com

Our Commitment to Building A Diverse and Inclusive Workforce

As a global and multi-cultural organization with varied ethnicities thriving across locations, we realize that our responsibility towards fulfilling the D&I commitment is huge. Therefore, we continuously strive to create a diverse, inclusive, and safe environment, for all our people, communities, and customers. Our leaders are committed to create an inclusive work culture which enables transparency, flexibility, and unbiased attention to every PayUneer so they can succeed, irrespective of gender, color, or personal faith. An environment where every person feels they belong, that they are listened to, and where they are empowered to speak up. At PayU we have zero tolerance towards any form of prejudice whether a specific race, ethnicity, or of persons with disabilities, or the LGBTQ communities.