Lead Software Engineer - Pyspark, AWS, Python

JPMorganChase; Maharashtra; On-site; INR 4,000,000 - 7,000,000; Full time

Job summary

JPMorganChase is seeking a Lead Software Engineer to drive data engineering initiatives in a fast-paced, cloud-first environment.

You will lead design and implementation of scalable data pipelines, ensure data governance, and collaborate with cross-functional teams to deliver high-impact analytics solutions. The role emphasizes AI-assisted engineering practices and secure, reliable data systems.

Qualifications

• 5+ years of software engineering experience in data-heavy environments.
• Strong command of distributed data processing frameworks and cloud data lakehouse platforms.
• Proficiency with Python/Java/Scala, SQL, and data serialization formats.

Responsibilities

• Lead design, development, and maintenance of cloud-based data pipelines and infrastructure.
• Architect and refine data models for large-scale datasets and analytics.
• Partner with cross-functional teams to translate requirements into scalable data solutions.
• Drive data quality, governance, and regulatory compliance across the data stack.
• Define data strategy and end-to-end data infrastructure lifecycle.

Skills

Spark

Python

Java/Scala

SQL

Data modeling

CI/CD

TDD/BDD

AI-assisted development

Data governance

Kafka

AWS Cloud

Education

Bachelor's degree in Computer Science

Tools

Spark

Airflow

Docker

Kubernetes

Snowflake

Job description

Job Description

We have an opportunity to impact your career and provide an adventure where you can push the limits of what's possible.

As a Lead Software Engineer at JPMorganChase within the Consumer and Community Banking, you are an integral part of an agile team that works to enhance, build, and deliver trusted market-leading technology products in a secure, stable, and scalable way. As a core technical contributor, you are responsible for conducting critical technology solutions across multiple technical areas within various business functions in support of the firm's business objectives.

Job Responsibilities

• Lead the design, development, and maintenance of robust, scalable cloud-based data processing pipelines and infrastructure, ensuring adherence to engineering standards, governance frameworks, and industry best practices.
• Architect and refine data models for large-scale datasets, optimizing for efficient storage, high-performance retrieval, and advanced analytics while upholding data integrity and quality.
• Partner with cross-functional teams to translate complex business requirements into effective, scalable data engineering solutions that drive organizational value.
• Champion a culture of innovation and continuous improvement, proactively identifying and implementing enhancements to data infrastructure, processing workflows, and analytics capabilities.
• Define and execute data strategy, including the development of enterprise data models and the management of end-to-end data infrastructure—from design and construction to installation and ongoing maintenance of large-scale processing systems.
• Drive data quality initiatives, ensure seamless data accessibility for analysts and data scientists, and maintain strict compliance with data governance and regulatory requirements.
• Align data engineering practices with business objectives, ensuring solutions are both technically sound and strategically relevant.
• Author, review, and approve technical requirements and architectural designs, and lead process re-engineering efforts to deliver cost-effective, high-impact business solution
• Drives team adoption of enterprise-authorized AI-assisted engineering practices within the work environment to improve code quality, delivery speed, and operational outcomes (e.g., AI-assisted code review/refactoring, test strategy acceleration, incident/root-cause analysis support), while establishing consistent validation standards (secure coding, peer review, automated testing) and promoting reuse of effective patterns across the team.
• Applies knowledge of tools within the Software Development Life Cycle toolchain, including enterprise-authorized AI-assisted development and automation capabilities, to improve the value realized by automation.

Required Qualifications, Capabilities, And Skills

• Formal training or certification on software engineering concepts and 5+ years applied experience
• Expert in at least one distributed data processing framework (Spark). Expert in at least one cloud data Lakehouse platforms (AWS Data lake services or Databricks, if not Hadoop),
• Expert in at least one scheduling/orchestration tools ( Airflow, alternatively AWS Step Functions or similar) & Expert with relational and NoSQL databases. Expert in data structures, data serialization formats (JSON, AVRO, Protobuf, or similar), and big-data storage formats (Parquet, Iceberg, or similar)
• Hands‑on professional experience in one or more programming language(s), including Java or Python, proficiency in Python, SQL, and at least one additional language (e.g. Java or Scala) for data engineering tasks
• Hands‑on experience utilizing Apache Spark for large-scale data processing, including developing and optimizing data pipelines, performing real‑time and batch analytics, and leveraging Spark's libraries for machine learning and data transformation to drive actionable business insights.
• Proficiency in microservices architecture, serverless computing and distributed cluster computing tools such as Docker, Kubernetes etc. Experience in one or more data modelling techniques (Dimensional, Data Vault, Kimball, Inmon, etc.)
• Experience with test‑driven development (TDD) or behavior‑driven development (BDD) practices, as well as working with continuous integration and continuous deployment (CI/CD) tools.
• Experience organizing and leading design workshops, coding sessions, and hackathons to promote a culture of excellence and innovation in data engineering. Expertise in architecting reusable, future‑ready design patterns that address diverse use cases across the organization.
• Expertise in working with streaming platforms like Kafka, MQ etc.
• Demonstrated experience leading effective use of approved AI‑assisted software development tools (e.g., for coding, code review, test acceleration, troubleshooting) with the ability to set team expectations for validating AI outputs for correctness, performance, and security.
• Strong understanding of responsible AI use in engineering workflows, including data sensitivity considerations, secure handling of inputs/outputs, and adherence to resiliency and security expectations; experience coaching engineers on safe, compliant adoption within delivery practices

Preferred Qualifications, Capabilities, And Skills

• Hands‑on experience with Infrastructure as Code (IaC) tools, preferably Terraform; experience with AWS CloudFormation is also valued.
• Proficiency in cloud‑based data pipeline technologies such as Spinnaker or similar platforms.
• Strong working knowledge of the Snowflake data platform.
• Experience in budgeting and resource allocation for data engineering projects.
• Proven ability to manage vendor relationships effectively.