Lead Software Engineer - Data & AI Platform

JPMorgan Chase & Co.; Jersey City (NJ); On-site; USD 61,000 - 101,000; Full time

Job summary

JPMorganChase in Jersey City seeks a senior software/data platform engineer to design, build, and operate scalable data pipelines and lakehouse capabilities using Spark, Airflow, Kafka, Flink, and Kubernetes.

You will apply data modeling, CI/CD, and governance practices, work with Databricks, AWS services, and agentic AI topics, and partner with analytics teams to deliver production-ready solutions.

Qualifications

• Formal training or certification in software engineering concepts.
• 5+ years of hands-on software/data platform experience.
• Experience delivering system design, application development, testing, and operational stability.
• Proven experience in software engineering or data platform development.
• Advanced proficiency in Python, Java, and SQL.
• Hands-on experience with Spark, Flink, and related distributed processing frameworks.
• Strong data modeling skills (star/snowflake schemas) and query optimization.
• Experience building data pipelines on Databricks using Airflow.
• Proficiency with cloud data services (AWS S3/Glue/Redshift/Athena/EMR, Lake Formation).
• Experience building production-grade data platforms on Kubernetes with catalog integration (Iceberg,Unity Catalog, OpenMetadata).
• Understanding of CI/CD, application resiliency, and security.
• Experience developing Agentic AI, LLMs, RAG, MCP, vector databases, and embedding-based retrieval.

Responsibilities

• Design, build, and maintain scalable data pipelines and ETL/ELT workflows for batch and real-time processing using Spark, Airflow, Kafka, and Flink.
• Develop data platform capabilities including data cataloging, data quality, governance, lineage, and compliance.
• Implement data modeling strategies to support analytics, reporting, and downstream use.
• Partner with analytics teams, product managers, and stakeholders to deliver production-ready data solutions.
• Produce secure, high-quality production code; review and debug others' code.
• Identify and automate remediation to strengthen operational stability across apps and systems.
• Lead evaluation sessions with vendors and teams to assess architectures and fit within information architecture.
• Drive development of Agentic Autonomous Lakehouse capabilities and governance automation.
• Lead communities of practice to promote adoption of new technologies.

Skills

Python

Java

SQL

CI/CD

Agile

Data modeling

Distributed processing

Tools

Apache Spark

Flink

Databricks

Airflow

Docker

Kubernetes

Terraform

Iceberg/Open metadata

Job description

Salary: $61,000 - 101,000 per year

Requirements:

• We need formal training or certification in software engineering concepts, plus 5+ years of hands‑on experience.
• We require practical experience delivering system design, application development, testing, and operational stability.
• We look for proven professional experience in software engineering or data platform development.
• We need advanced proficiency in one or more programming languages, including Python, Java, and SQL.
• We require hands‑on experience with distributed data processing frameworks such as Apache Spark and Flink.
• We need a strong grasp of data modeling approaches, including star schema and snowflake, as well as query optimization.
• We require experience designing and running data pipelines on Databricks using orchestration tools such as Apache Airflow.
• We need proficiency with cloud data services such as AWS S3, Glue, Redshift, Athena, EMR, and Lake Formation, or similar platforms.
• We require experience building production‑grade data platforms on Kubernetes with open catalog integration, such as Apache Iceberg, Unity Catalog, or OpenMetadata, to support discovery, lineage, and governance.
• We need an advanced understanding of agile practices, including CI/CD, application resiliency, and security.
• We require experience developing Agentic AI, LLMs, RAG architectures, MCP, vector databases, and embedding‑based retrieval systems.
• Preferred: familiarity with data platform and transformation framework development.
• Preferred: experience with data mesh or data product architectures.
• Preferred: proficiency with Infrastructure as Code using Terraform and containerized deployments with Docker and Kubernetes.
• Preferred: experience with data observability, quality, and metadata management tools.
• Preferred: experience with semantic layers, metrics stores, or BI platforms such as Tableau and dbt Metrics.

Responsibilities:

• We execute creative software design, development, and technical troubleshooting, approaching problems beyond standard or conventional methods.
• We design, build, and maintain scalable data pipelines and ETL/ELT workflows for both batch and real‑time processing using Spark, Airflow, Kafka, and Flink.
• We develop data platform capabilities such as data cataloging, data quality frameworks, and semantic or metrics layers with governance, lineage, and compliance built in.
• We implement data modeling strategies, including fact and dimensional models and wide tables, to support analytics, reporting, and downstream use.
• We partner with analytics teams, product managers, and business stakeholders to turn data needs into production‑ready solutions.
• We produce secure, high‑quality production code and review and debug code written by others.
• We identify opportunities to remove or automate remediation of recurring issues to strengthen operational stability across applications and systems.
• We lead evaluation sessions with vendors, startups, and internal teams to assess architectural designs, technical strengths, and fit within our systems and information architecture.
• We lead development of the Agentic Autonomous Lakehouse capability, automating governed self‑service pipeline provisioning and lakehouse operations such as health, cost, performance analysis and best‑practice enforcement.
• We lead communities of practice across Software Engineering to promote awareness and adoption of emerging and leading‑edge technologies.
• We contribute to a team culture rooted in diversity, opportunity, inclusion, and respect.

Technologies:

• Agentic AI
• AI
• Airflow
• AWS
• Redshift
• CI/CD
• Cloud
• Databricks
• Docker
• ETL
• Flink
• Support
• Java
• Kafka
• Kubernetes
• MCP
• Python
• RAG
• SQL
• Security
• Snowflake
• Spark
• Tableau
• Terraform
• Unity
• dbt
• GameDev
• Embedded

More:

We are JPMorganChase, one of the worlds oldest financial institutions, delivering innovative financial solutions to millions of consumers, small businesses, and leading corporate, institutional, and government clients under the J.P. Morgan and Chase brands. Our history spans more than 200 years, and we are a global leader in investment banking, consumer and small business banking, commercial banking, financial transaction processing, and asset management. We offer a competitive total rewards package with base salary based on role, experience, skills, and location, and eligible roles may also include commission or discretionary incentive compensation. Our benefits and programs may include comprehensive health coverage, on‑site health and wellness centers, retirement savings, backup childcare, tuition reimbursement, mental health support, financial coaching, and more. We value diversity and inclusion, support reasonable accommodations, and are part of a global Commercial & Investment Bank that operates across banking, markets, securities services, and payments in more than 100 countries.

last updated 40 week of 2026