Senior Databricks PySpark AI Engineer

JPMorgan Chase & Co.; Wilmington (DE); On-site; USD 140,000 - 210,000; Full time

Job summary

JPMorgan Chase & Co. is seeking a Lead Software Engineer specializing in Databricks/PySpark/AI to drive data engineering across the Global Finance team.

You will mentor engineers, deliver production‑grade data products, and optimize pipelines that power autonomous AI agents within a robust enterprise data infrastructure. You will collaborate with product owners and data scientists to translate business requirements into scalable AI solutions, using AWS services and state‑of‑the‑art data tooling

Qualifications

• 5+ years practical software engineering experience.
• Experience leading AI-assisted software development tools and code reviews.
• Strong understanding of responsible AI in engineering workflows.
• Expert-level Python/PySpark production-grade coding.
• Extensive hands-on Databricks and AWS cloud ecosystem experience (Glue, S3, SQS/SNS, Lambda, Spark, SQL).
• Deep knowledge of Big Data and data warehousing concepts at enterprise scale.
• Experience with CI/CD and automated testing frameworks.
• Solid grasp of Agile methodologies, DevOps, security and resiliency.
• Understanding of agentic AI concepts and data infrastructure to support them.
• Experience building APIs and data services for AI agents.

Responsibilities

• Build and optimize data pipelines and workflows powering agentic AI systems with reliable, real-time data access.
• Drive enterprise AI-assisted engineering practices to improve code quality, delivery speed and operational outcomes.
• Apply SDLC tools to enhance automation and value realization.
• Develop data retrieval and indexing layers enabling autonomous search across multiple data sources.
• Create data services and APIs that AI agents invoke to access enterprise data.
• Enforce data governance, quality, security, and compliance for AI- driven data assets.
• Develop production-grade, secure code and deploy via CI/CD pipelines.
• Implement memory and state management for multi-step agent workflows.
• Mentor junior engineers through pairing, code reviews and coaching.
• Collaborate with product owners, data scientists and stakeholders to translate requirements into production AI solutions.
• Evaluate emerging AI frameworks and data engineering practices to improve capabilities.

Skills

Python/PySpark

Databricks

AWS Cloud

CI/CD

Agile methodologies

LLMs

Security/compliance

Tools

Snowflake

Terraform

Job description

JPMorgan Chase & Co. is seeking a Lead Software Engineer specializing in Databricks/PySpark/AI to drive data engineering across the Global Finance team.

You will mentor engineers, deliver production‑grade data products, and optimize pipelines that power autonomous AI agents within a robust enterprise data infrastructure. You will collaborate with product owners and data scientists to translate business requirements into scalable AI solutions, using AWS services and state‑of‑the‑art data tooling