Lead Data Engineer

JPMorganChase; Pune District; On-site; INR 1,500,000 - 2,500,000; Full time

Job summary

A leading global financial services firm is seeking a Lead Data Engineer to manage scalable data processing pipelines and enhance data infrastructures. This pivotal role involves working collaboratively to address complex data challenges and drive organizational value through innovative data solutions. Ideal candidates will possess expertise in programming languages like Python and Java, as well as cloud platforms, ensuring robust, compliant, and quality data management processes. This role is based in Pune, Maharashtra, and requires substantial technical proficiency and leadership in data engineering.

Qualifications

• Expert in at least two programming languages like Python and Java.
• Experience with distributed data processing frameworks like Spark.
• Proficient in cloud data Lakehouse platforms such as AWS or Databricks.

Responsibilities

• Lead the development of scalable cloud-based data processing pipelines.
• Architect data models for efficient storage and retrieval.
• Drive data quality initiatives and ensure data governance compliance.

Skills

Python

Java

Spark

AWS Data Lake Services

Airflow

Microservices Architecture

Docker

Kubernetes

Data Modelling Techniques

Tools

Terraform

Snowflake

Job description

Lead Data Engineer

Job Description

Embrace this pivotal role as an essential member of a high performing team dedicated to reaching new heights in data engineering. Your contributions will be instrumental in shaping the future of one of the world’s largest and most influential companies.

As a Lead Data Engineer at JPMorgan Chase within the Consumer & Community Banking in Connected Commerce Travel Technology Team, you are an integral part of an agile team that works to enhance, build, and deliver large-scale data collection, storage, access, and analytics in a secure, stable, and scalable way.

Leverage your deep technical expertise and problem solving capabilities to lead and drive significant business impact and tackle a diverse array of challenges that span multiple data pipelines, data architectures, and other data consumers.

Job Responsibilities

• Lead the design, development, and maintenance of robust, scalable cloud-based data processing pipelines and infrastructure, ensuring adherence to engineering standards, governance frameworks, and industry best practices.
• Architect and refine data models for large-scale datasets, optimizing for efficient storage, high-performance retrieval, and advanced analytics while upholding data integrity and quality.
• Partner with cross-functional teams to translate complex business requirements into effective, scalable data engineering solutions that drive organizational value.
• Champion a culture of innovation and continuous improvement, proactively identifying and implementing enhancements to data infrastructure, processing workflows, and analytics capabilities.
• Define and execute data strategy, including the development of enterprise data models and the management of end-to-end data infrastructure—from design and construction to installation and ongoing maintenance of large-scale processing systems.
• Drive data quality initiatives, ensure seamless data accessibility for analysts and data scientists, and maintain strict compliance with data governance and regulatory requirements.
• Align data engineering practices with business objectives, ensuring solutions are both technically sound and strategically relevant.
• Author, review, and approve technical requirements and architectural designs, and lead process re-engineering efforts to deliver cost-effective, high-impact business solution.

Required Qualifications, Capabilities, And Skills

• Expert in at least two programming languages (Python, and Java)
• Expert in at least one distributed data processing framework (Spark)
• Expert in at least one cloud data Lakehouse platforms (AWS Data lake services or Databricks, if not Hadoop)
• Expert in at least one scheduling/orchestration tools (Airflow, alternatively AWS Step Functions or similar)
• Expert with relational and NoSQL databases.
• Expert in data structures, data serialization formats (JSON, AVRO, Protobuf, or similar), and big-data storage formats (Parquet, Iceberg, or similar)
• Proficiency in microservices architecture, serverless computing and distributed cluster computing tools such as Docker, Kubernetes etc.
• Proficiency in one or more data modelling techniques (Dimensional, Data Vault, Kimball, Inmon, etc.)
• Experience with test-driven development (TDD) or behavior-driven development (BDD) practices, as well as working with continuous integration and continuous deployment (CI/CD) tools.
• Experience organizing and leading design workshops, coding sessions, and hackathons to promote a culture of excellence and innovation in data engineering.
• Expertise in architecting reusable, future-ready design patterns that address diverse use cases across the organization.

Preferred Qualifications, Capabilities, And Skills

• Hands-on experience with Infrastructure as Code (IaC) tools, preferably Terraform; experience with AWS CloudFormation is also valued.
• Proficiency in cloud-based data pipeline technologies such as Fivetran, dbt, Prophecy.io, or similar platforms.
• Experience with front-end development frameworks, ideally React; familiarity with Angular is also beneficial.
• Strong working knowledge of the Snowflake data platform.
• Experience in budgeting and resource allocation for data engineering projects.
• Proven ability to manage vendor relationships effectively.