Senior Data Engineer

IDFC FIRST Bank; Bengaluru; On-site; INR 2,500,000 - 4,000,000; Full time

Job summary

IDFC FIRST Bank is seeking a Senior Data Engineer in Bengaluru to design, build, and maintain scalable data pipelines and data lake architectures. You will lead engineering initiatives, collaborate with cross-functional teams, and shape enterprise-wide data strategy and innovation.

You will work with SQL, Spark, Python, Scala, and cloud-based data platforms to deliver high-availability data solutions and APIs for analytics and BI.

Qualifications

• Bachelor’s degree in computer science, information technology or engineering.
• 6+ years of experience in data engineering, with at least 3 years in large-scale Data Lake ecosystems.

Responsibilities

• Design and develop scalable data pipelines across the full data lifecycle—from ingestion and transformation to consumption.
• Work with large-scale data volumes (TBs/PBs) and ensure efficient data processing and storage.
• Collaborate with business stakeholders to identify high-impact problems and translate them into data solutions.
• Apply advanced knowledge of SQL, Spark, Python, Scala, and the Hadoop ecosystem.
• Implement and manage Data Lake/Lakehouse architectures using platforms like Cloudera, Hortonworks, and AWS.
• Develop APIs and integrate them into data workflows for seamless data access and usability.
• Utilize big data infrastructure tools including Spark Streaming, EMR, MapReduce, Hive, HDFS, YARN, HBase, and Oozie.
• Create and maintain technical design documentation (HLD/LLD) for data pipelines and architecture.
• Debug and resolve technical issues in data architecture and ensure high availability and performance.

Skills

SQL

Spark

Python

Scala

Hadoop ecosystem

Education

Bachelor's degree in computer science or information technology or engineering

Tools

Cloudera

Hortonworks

AWS

EMR

MapReduce

Hive

HDFS

YARN

HBase

Oozie

Job description

Job Requirements

About the Role

As a Senior Data Engineer in the New Age Data & Analytics team at IDFC FIRST Bank, you will be responsible for designing, building, and maintaining scalable data pipelines and infrastructure to support advanced analytics and business intelligence. This role demands deep technical expertise in big data technologies, cloud platforms, and data architecture. You will also lead engineering initiatives, collaborate with cross-functional teams, and contribute to enterprise-wide data strategy and innovation.

Key Responsibilities

Primary Responsibilities

• Design and develop scalable data pipelines across the full data lifecycle—from ingestion and transformation to consumption.
• Work with large-scale data volumes (TBs/PBs) and ensure efficient data processing and storage.
• Collaborate with business stakeholders to identify high-impact problems and translate them into data solutions.
• Apply advanced knowledge of SQL, Spark, Python, Scala, and the Hadoop ecosystem.
• Implement and manage Data Lake/Lakehouse architectures using platforms like Cloudera, Hortonworks, and AWS.
• Develop APIs and integrate them into data workflows for seamless data access and usability.
• Utilize big data infrastructure tools including Spark Streaming, EMR, MapReduce, Hive, HDFS, YARN, HBase, and Oozie.
• Create and maintain technical design documentation (HLD/LLD) for data pipelines and architecture.
• Debug and resolve technical issues in data architecture and ensure high availability and performance.

Secondary Responsibilities

• Work independently and manage your own development efforts.
• Collaborate with BI analysts and engineers to prototype and implement analytical models.
• Identify key performance indicators and define strategies to meet analytical goals.
• Present data solutions and insights to stakeholders and leadership teams.

Managerial & Leadership Responsibilities

• Lead moderately complex data engineering initiatives aligned with enterprise strategy.
• Build and maintain optimized, highly available data pipelines to support analytics and reporting.
• Oversee data integration efforts, including modeling, warehousing, and analytics environments.
• Resolve complex data engineering issues while ensuring compliance with data governance policies.
• Collaborate with peers and managers to align on strategic goals and deliverables.

What We Are Looking For

Education

• Bachelor’s degree in computer science or information technology or engineering

Experience

• 6+ years of experience in data engineering, with at least 3 years in large-scale Data Lake ecosystems.

Skills and Attributes

• Proven expertise in SQL, Spark, Python, Scala, and big data tools.
• Strong understanding of Data Lake/Lakehouse architecture and cloud platforms (especially AWS).
• Experience with machine learning frameworks and real-time data processing.
• Deep knowledge of relational and non-relational databases and data mining techniques.
• Strong debugging and problem-solving skills.
• Excellent communication and documentation abilities.
• Ability to lead technical projects and mentor junior engineers.

Key Success Metrics

• Timely and error-free delivery of data engineering projects.
• Identification and resolution of data issues.
• Leadership in technical aspects of data initiatives.
• High performance and availability of data pipelines.