Senior Databricks Spark Engineer - AWS Data Lakehouse

Next Frontier Capital; Kentucky; On-site; USD 150,000 - 190,000; Full time

Benefits offered by this job

Health care coverage

On-site health centers

Retirement savings plan

Tuition reimbursement

Mental health support

Job summary

JPMorganChase is seeking a Software Engineer III to join the Employee and Experience Technology team. You will design and deliver trusted technology products, building Databricks-on-AWS lakehouse data pipelines with PySpark at scale.

You will implement data quality, monitoring, AI-assisted tooling, and robust automation across workload pipelines, partnering with BI teams using Sigma, Tableau, and Alteryx. Emphasis on security, stability, and agile delivery is core to the role.

Qualifications

• Formal training or certification in software engineering concepts with 3+ years of applied experience in data engineering.
• Advanced, hands-on expertise in Apache Spark (PySpark) for large-scale distributed data processing on Databricks.
• Strong expertise across the AWS data ecosystem including S3, EMR, Glue, Lambda, and Athena.
• Strong programming skills in Python for data processing and application development.
• Proficiency in automation and CI/CD pipelines with version control tooling.
• Hands-on experience delivering design, development, testing, and operational stability with agile methods.
• In-depth knowledge of financial services IT systems and data security.

Responsibilities

• Design, build, and maintain new data pipelines on Databricks using PySpark.
• Optimize PySpark jobs and Databricks clusters for performance and cost efficiency.
• Design scalable data frameworks for end-to-end Databricks pipelines for analytics.
• Implement data quality checks using Delta Lake and Delta Live Tables.
• Implement monitoring and alerting with Databricks and AWS CloudWatch.
• Automate remediation and improve operational stability with Databricks Workflows.
• Leverage AI tools to accelerate data pipeline development and quality.
• Deliver curated data sets to BI partners (Sigma, Tableau, Alteryx).
• Collaborate with stakeholders to design architectures and artifacts.
• Contribute to practice communities and promote inclusion.

Skills

Apache Spark (PySpark)

Databricks

AWS data ecosystem

Python

CI/CD

Airflow

Autosys

SQL & data modeling

Delta Lake

AI tooling

Tools

Databricks

Git/Bitbucket

Jenkins

Spinnaker

Airflow

Delta Lake

Claude

GitHub Copilot

Job description

JPMorganChase is seeking a Software Engineer III to join the Employee and Experience Technology team. You will design and deliver trusted technology products, building Databricks-on-AWS lakehouse data pipelines with PySpark at scale.

You will implement data quality, monitoring, AI-assisted tooling, and robust automation across workload pipelines, partnering with BI teams using Sigma, Tableau, and Alteryx. Emphasis on security, stability, and agile delivery is core to the role.