Senior Data Engineer

AU SMALL FINANCE BANK; Mumbai; On-site; INR 4,000,000 - 9,000,000; Full time

Job summary

AU SMALL FINANCE BANK is seeking a Senior Data Engineer with 7-10 years of experience to design, build, and manage enterprise-scale data platforms and data warehouses. You will architect scalable ETL/ELT pipelines on AWS and lead delivery of secure, high-performance data ecosystems.

The role requires expertise in data warehousing, big data, AWS data services, data modeling, performance optimization, and people leadership in a fast-paced banking environment.

Qualifications

• 7+ years of overall Data Engineering experience.
• 7+ years in Enterprise Data Warehousing.
• Hands-on ETL/ELT pipelines using Spark, Scala, and Python.
• Strong SQL development and performance tuning.
• Experience in Data Modeling, Data Architecture, and Data System Design.
• Extensive experience with AWS Data Services (EMR, Redshift, S3, Athena, Glue, Airflow).
• Experience supporting production environments and handling critical incidents.
• Exposure to banking/financial services or large enterprises preferred.

Responsibilities

• Design, develop, and maintain scalable data pipelines for batch and real-time processing.
• Architect end-to-end ETL/ELT solutions across enterprise systems.
• Build and optimize cloud-native data platforms using AWS services.
• Define data strategies including sourcing, flow, storage, governance, and consumption.
• Collaborate with analytics, BI, product, and technology teams to meet data requirements.
• Design and implement scalable data models supporting reporting and BI.
• Monitor, troubleshoot, and support production data pipelines for high availability.
• Lead performance tuning for SQL queries and large-scale workloads.
• Mentor junior engineers and coordinate project tasks.
• Prepare architecture diagrams and runbooks.

Skills

Data Engineering

Data Warehousing

ETL/ELT

Python

Scala

Spark

SQL

Data Modeling

Mentoring

Communication

Education

BE/B.Tech/M.Tech

Tools

AWS (S3, EMR, Redshift)

Apache Airflow

AWS Glue

Amazon S3

Amazon Redshift

Amazon EMR

Amazon Athena

Spark

Python

SQL

Job description

Job Description

Job Summary

We are seeking a highly skilled Senior Data Engineer with 7-10 years of experience in designing, developing, and managing enterprise-scale data platforms and data warehousing solutions. The ideal candidate will have extensive experience in building scalable ETL/ELT pipelines, architecting cloud-based data solutions on AWS, and leading teams in delivering reliable, secure, and high-performance data ecosystems.

The role requires strong expertise in Data Warehousing, Big Data technologies, AWS Data Services, Data Modeling, Performance Optimization, and People Management within a fast-paced banking and financial services environment.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines for batch and real-time data processing.
• Architect end-to-end ETL/ELT solutions for data ingestion, transformation, and consumption across enterprise systems.
• Build and optimize cloud-native data platforms using AWS services such as S3, EMR, Redshift, Glue, Athena, and Airflow.
• Define and implement enterprise data strategies, including data sourcing, data flow, storage, governance, and consumption frameworks.
• Collaborate with Data Analytics, Business Intelligence, Product, and Technology teams to understand and fulfill data requirements.
• Design and implement scalable data models supporting reporting, analytics, and business intelligence initiatives.
• Monitor, troubleshoot, and support production data pipelines to ensure high availability and reliability.
• Lead performance tuning initiatives for complex SQL queries, ETL jobs, and large-scale data processing workloads.
• Ensure data quality, validation, governance, security, and compliance standards are adhered to.
• Drive cloud cost optimization initiatives while maintaining performance and user experience.
• Present technical architecture, solutions, and recommendations to business and technology stakeholders.
• Mentor junior engineers and coordinate tasks across project teams.
• Prepare and maintain technical documentation, architecture diagrams, and operational runbooks.

Required Technical Skills

Data Engineering & Data Warehousing

• Enterprise Data Warehousing Concepts
• Data Lake & Data Lakehouse Architecture
• ETL/ELT Design and Development
• Data Integration and Data Migration
• Data Quality & Validation Frameworks

Programming & Big Data

• Python
• Scala
• Apache Spark (PySpark/Spark SQL)
• SQL Query Optimization
• Distributed Data Processing

AWS Data Stack

• Amazon S3
• Amazon EMR
• Amazon Redshift
• AWS Glue
• Amazon Athena
• Apache Airflow
• IAM & AWS Security Best Practices

Data Modeling & Architecture

• Dimensional Data Modeling
• Star Schema & Snowflake Schema
• Data Architecture Design
• Metadata Management
• Data Governance Frameworks

Scheduling & Orchestration

• Apache Airflow
• Enterprise Job Scheduling Frameworks
• Workflow Automation

Monitoring & Production Support

• Data Pipeline Monitoring
• Incident Management
• Root Cause Analysis
• SLA Management

Security

• Data Security Principles
• Data Encryption & Access Controls
• Regulatory and Compliance Awareness

Experience Requirements

• Minimum 7+ years of overall Data Engineering experience .
• Strong experience in Enterprise Data Warehousing (7+ years) .
• Hands-on expertise in building scalable ETL pipelines using Spark, Scala, and Python (7+ years) .
• Strong SQL development and performance tuning experience.
• Experience in Data Modeling, Data Architecture, and Data System Design.
• Extensive experience working with AWS Data Services including EMR, Redshift, S3, Athena, Glue, and Airflow.
• Experience supporting production environments and handling critical incidents.
• Exposure to banking, financial services, fintech, or large enterprise environments preferred.

Leadership & Soft Skills

• Experience leading and mentoring data engineering teams.
• Strong stakeholder management and communication skills.
• Ability to present complex technical solutions to leadership and cross-functional teams.
• Strong analytical and problem-solving capabilities.
• Excellent documentation and technical writing skills.
• Ability to manage multiple priorities and deliver under tight timelines.

Preferred Qualifications

• BE/B.Tech/M.Tech from reputed Tier-1 Institutes.
• AWS Certifications (Solutions Architect, Data Engineer, Developer Associate, etc.).
• Certifications in Spark, Big Data, or Cloud Technologies will be an added advantage.