Lead Software Engineer - Data Engg. - Databricks / Snowflake

JPMorgan Chase & Co.; Plano (TX); On-site; USD 140,000 - 200,000; Full time

Job summary

JPMorgan Chase & Co. in Plano, TX seeks a Lead Software Engineer to design, build, and operate scalable data processing solutions using Databricks, Python, and AWS.

You will own data pipelines and a control plane for enterprise workflows, drive observability, security-by-design, and AI-assisted development while mentoring engineers across the SDLC.

Collaborate with cross-functional teams to deliver reliable data services that support business objectives and regulatory requirements.

Qualifications

• Formal training or certification in software engineering concepts with 5+ years of applied experience.
• Experience in data management and large-scale ETL/ELT processing with strong SQL, Python, and PySpark performance tuning.
• Hands-on Databricks/Spark experience and cloud data lake patterns, AWS integration.
• Proven experience building platform services/control planes including API design and config-driven systems.
• Strong focus on data quality, security-by-design, and lineage/auditability (IAM, secrets mgmt).
• Production engineering mindset: observability, monitoring, and incident response for always-on services.
• Proficient in CI/CD and release engineering with firm-standard tooling.
• Experience leading the adoption of AI-assisted development tools with governance.

Responsibilities

• Execute creative, data-driven software solutions end-to-end, solving complex technical problems.
• Design and build a control plane for enterprise data pipelines, standardizing orchestration and governance.
• Develop self-service APIs/SDKs, templates, and configuration-driven onboarding with guardrails.
• Design, develop, and maintain scalable data pipelines and processing workflows using Python, PySpark, SQL, Databricks on AWS.
• Ensure data quality, security, lineage, and observability with dashboards and automated remediation patterns.
• Lead and participate in the full SDLC, providing production support for pipeline and platform services.
• Collaborate with stakeholders to shape data management strategy and translate requirements into scalable solutions.
• Mentor engineers and foster adoption of modern engineering practices, including AI-assisted tools.
• Drive enterprise-wide AI-assisted engineering practices with validation standards and reuse of patterns.
• Apply SDLC toolchain knowledge to improve automation and value.

Skills

Data management

ETL/ELT

Python

PySpark

AI-assisted development

Security-by-design

Observability

Tools

Databricks

Spark

AWS

Terraform/CloudFormation

Jenkins

Spinnaker

Sonar

Airflow

Job description

We have an opportunity to impact your career and provide an adventure where you can push the limits of what's possible.

As a Lead Software Engineer at JPMorgan Chase within the (IAM) Identity and Access Management Data team, you will play a crucial role in designing, developing, and maintaining scalable data processing solutions using Databricks, Python, and AWS. You will collaborate with cross-functional teams to deliver high-quality data solutions that support our business objectives.

Job responsibilities

• Execute creative, data-driven software solutions end-to-end (design, development, troubleshooting), thinking beyond routine approaches to solve complex technical problems.
• Design and build acontrol plane for enterprise data pipelines, standardizing pipeline definition, scheduling, deployment, governance, and run-time management (Databricks today; extensible for future engines).
• Develop self-serviceAPIs/SDKs, templates, and configuration-driven onboarding with consistent guardrails (standards, validation, environment promotion, approvals) and centralized pipeline metadata (ownership, SLAs/SLOs, dependencies, schema/parameter/version tracking).
• Design, develop, and maintain scalabledata pipelines and processing workflows using Python, PySpark, SQL, Databricks on AWS; develop fact/dimension models for analytics and reporting.
• Ensure data quality, security, lineage, and operational transparency via standardized observability (logs/metrics/traces), dashboards, alerting, runbooks, and automated remediation patterns (retries/backfills, common-failure automation).
• Lead and participate in the full SDLC (requirements, design, build, test, deploy, maintain), acting as SRE/production support for pipeline and platform services to improve stability and reliability.
• Collaborate with stakeholders to shape data management strategy and translate requirements into scalable, compliant solutions; document data flows, logic, and transformation rules for knowledge sharing.
• Mentor engineers and lead communities of practice to drive adoption of modern engineering practices and tools, fostering an inclusive, high-performing culture; utilize firm-approved AI-assisted development tools to accelerate delivery and testing
• Drives team adoption of enterprise-authorized AI-assisted engineering practices within the work environment to improve code quality, delivery speed, and operational outcomes (e.g., AI-assisted code review/refactoring, test strategy acceleration, incident/root-cause analysis support), while establishing consistent validation standards (secure coding, peer review, automated testing) and promoting reuse of effective patterns across the team.
• Applies knowledge of tools within the Software Development Life Cycle toolchain, including enterprise-authorized AI-assisted development and automation capabilities, to improve the value realized by automation.

Required qualifications, capabilities, and skills

• Formal training or certification on software engineering concepts and 5+ years applied experience
• Proven experience indata management and ETL/ELT for large-scale processing, including strong SQL, Python, and PySpark with performance tuning and query optimization.
• Hands‑on experience withDatabricks/Spark and cloud data lake patterns, integrating compute/workflows with AWS services (e.g., S3, ECS, SNS/SQS, Lambda).
• Proven experience buildingplatform services/control planes (or similar orchestration/automation platforms), including API/service design, configuration-driven systems, and versioning/backward compatibility.
• Strong understanding ofdata quality, security-by-design, and lineage/auditability, including IAM/least privilege and secrets management principles.
• Strong production engineering mindset:observability (logs/metrics/traces), monitoring/alerting, incident response, and operational excellence for always-on services.
• Proficiency inCI/CD and release engineering (quality gates, automated testing, safe deployments/rollbacks) using firm-standard tooling (e.g., Jenkins/Jules, Spinnaker, Sonar).
• Demonstrated experience leading effective use of approved AI-assisted software development tools (e.g., for coding, code review, test acceleration, troubleshooting) with the ability to set team expectations for validating AI outputs for correctness, performance, and security.
• Strong understanding of responsible AI use in engineering workflows, including data sensitivity considerations, secure handling of inputs/outputs, and adherence to resiliency and security expectations; experience coaching engineers on safe, compliant adoption within delivery practices

Preferred qualifications, capabilities, and skills

• Experience with orchestration/execution frameworks (Databricks Workflows/Jobs, Airflow, Step Functions) and operational patterns such as dependency graphs (DAGs), replays, and backfills.
• Experience with data governance integrations (e.g., Unity Catalog concepts such as cataloging, permissions, and lineage hooks), where applicable.
• Infrastructure-as-Code experience (Terraform/CloudFormation) and developer-platform "golden path" enablement (internal CLIs, templates, paved roads, onboarding automation).
• Experience with FinOps/cost controls for Spark/Databricks workloads (telemetry, quotas, chargeback/showback) and data formats (Parquet, JSON, CSV, Avro, Delta Lake), Knowledge of regulatory reporting and financial data aggregation techniques; Databricks and/or AWS certifications.