Lead Data Engineer: AI-Driven Big Data & Data Mesh

Next Frontier Capital; Jersey City (NJ); On-site; USD 180,000 - 240,000; Full time

Job summary

JPMorganChase in Jersey City, NJ, seeks a Lead Software Engineer- Financial Service Data Engineering: PySpark/Java/BigData/Datalake/AI to join the Asset and Wealth Management–Global Prime Brokerage team. You will be a core technical contributor, building trusted data products and accelerating analytics delivery in a secure, scalable environment.

You will lead AI‑first development, define data product lifecycles, govern lineage and quality, and mentor teammates while partnering with product,

Qualifications

• Formal training or certification on software engineering concepts and 5+ years applied experience.
• Proven leadership delivering AI‑first analytics and data engineering at scale.
• Experience developing data ingestion and integration processes for analytics‑ready datasets.
• Deep hands‑on experience with big data platforms (Spark, Databricks, Snowflake, Iceberg) and building robust pipelines.
• Strong programming in Python and PySpark, or Java; CI/CD and containerization practices.
• Applied AI expertise in ML pipelines, NLP/LLMs, and autonomous agents for engineering tasks.
• Governance across lineage, data quality, and access control with regulatory alignment.
• Comfortable in agile environments; strong written and verbal communication.
• Experience leading AI-assisted software development tools and validation practices.
• Knowledge of responsible AI use in engineering workflows.

Responsibilities

• Collaborate with business and technology teams to develop AI‑first analytics and data product solutions.
• Define and enforce architecture for an AI‑driven data product lifecycle: semantic extraction/alignment, automated lineage and DQ, pipeline code generation, mesh registration, and self‑service consumption.
• Build and operate autonomous agents for data engineering that detect schema drift, propose transformations, reconcile semantics, triage data incidents, and maintain governance evidence.
• Design analytics platforms capable of running reporting and other analytics; explore real‑time and batch analytics.
• Establish monitoring and alerting of solution events related to performance, scalability, availability, and reliability.
• Provide technical leadership, guidance, and direction to other team members; build prototypes for demonstrations.
• Drive team adoption of enterprise‑authorized AI‑assisted engineering practices to improve code quality, delivery speed, and operational outcomes.
• Apply SDLC tooling to improve automation and value.
• Lead migration to the data mesh and modern stacks (Databricks/Iceberg/common services).
• Industrialize entity resolution and ML/LLM solutions to enable reuse and monetization.
• Embed governance, lineage, and DQ by design across critical domains and regulatory reporting.

Skills

PySpark

Python

Java

AI/ML pipelines

Data governance

Agile leadership

CI/CD

Data mesh knowledge

Tools

Spark

Databricks

Snowflake

Iceberg

Python

Java

Job description

JPMorganChase in Jersey City, NJ, seeks a Lead Software Engineer- Financial Service Data Engineering: PySpark/Java/BigData/Datalake/AI to join the Asset and Wealth Management–Global Prime Brokerage team. You will be a core technical contributor, building trusted data products and accelerating analytics delivery in a secure, scalable environment.

You will lead AI‑first development, define data product lifecycles, govern lineage and quality, and mentor teammates while partnering with product,