Lead Data Engineer — Agentic AI with PySpark & Databricks

Next Frontier Capital; Wilmington (DE); On-site; USD 150,000 - 210,000; Full time

Job summary

JPMorganChase is seeking a Lead Software Engineer - Databricks/PySpark/AI to drive data products for autonomous AI agents within Corporate Sector-Global Finance. You will lead hands-on development, build data infrastructure, and mentor engineers to raise engineering standards while delivering production-grade solutions.

The role emphasizes real-time data pipelines, secure coding practices, and cross-functional collaboration with product and data science teams to scale agentic AI capabilities

Qualifications

• Formal training or certification on software engineering concepts and 5+ years applied experience.
• Expert-level programming skills in Python/PySpark with production-grade code.
• Extensive hands-on experience with Databricks and AWS cloud services such as Glue, S3, SQS/SNS, Lambda.
• Deep expertise with Spark, SQL, and data warehousing at enterprise scale.
• Strong experience with Lakehouse/Delta Lake, testing, governance, and cost optimization.
• Familiarity with CI/CD processes and automated testing.

Responsibilities

• Build and optimize data pipelines and workflows for agentic AI systems requiring real-time data access.
• Develop data retrieval and indexing to enable autonomous search and synthesis across sources.
• Create and maintain tool-use infrastructure—APIs and data services—for enterprise data access by AI agents.
• Enforce data quality, security, governance, and compliance across autonomous data products.
• Deliver secure production code following AWS best practices and CI/CD deployments.
• Design memory, context persistence, and task chaining for multi-step agent workflows.
• Mentor junior engineers through pairing, code reviews, and technical coaching.
• Collaborate with product, data science, and business teams to translate requirements into production-ready solutions.

Skills

Python/PySpark

Databricks

AWS cloud

SQL

CI/CD pipelines

Mentoring

Agile

Data engineering

Big data

Tools

Terraform

Snowflake

OpenAI API

LangGraph

AutoGen

Job description

JPMorganChase is seeking a Lead Software Engineer - Databricks/PySpark/AI to drive data products for autonomous AI agents within Corporate Sector-Global Finance. You will lead hands-on development, build data infrastructure, and mentor engineers to raise engineering standards while delivering production-grade solutions.

The role emphasizes real-time data pipelines, secure coding practices, and cross-functional collaboration with product and data science teams to scale agentic AI capabilities