Senior Big Data Engineer: Spark Pipelines Lead

Next Frontier Capital; Plano (TX); On-site; USD 140,000 - 190,000; Full time

Job summary

JPMorganChase, Plano, TX, is seeking a Lead Software Engineer to design, automate, and operate scalable data transformation pipelines in production. You will work with Spark, HBase, Cassandra, and cloud tooling to deliver trusted data products in a secure, scalable fashion.

You will mentor engineers, contribute to platform infrastructure, and drive AI-assisted development practices, CI/CD, and observability with Grafana/Prometheus. The role emphasizes cost awareness and cross-team collaboration.

Qualifications

• 5+ years applied experience in software engineering.
• Advanced in one or more programming languages and Spark ecosystem (Python/Java/Scala).
• Experience designing and operating ETL pipelines with Spark, HBase, Cassandra, and cloud tooling.

Responsibilities

• Design, develop, and automate scalable data transformation pipelines for production environments.
• Mentor junior engineers and promote AI-assisted engineering practices.
• Lead CI/CD, monitoring (Grafana/Prometheus) and cost optimization for data platforms.
• Collaborate with stakeholders to gather requirements and deliver robust data solutions.

Skills

Apache Spark

Python

Java

Scala

Databricks

Spring Boot API

Docker

Grafana

Prometheus

CloudWatch

HBase

Cassandra

CI/CD

AI-assisted development

Tools

GitHub

Bitbucket

Confluence

Job description

JPMorganChase, Plano, TX, is seeking a Lead Software Engineer to design, automate, and operate scalable data transformation pipelines in production. You will work with Spark, HBase, Cassandra, and cloud tooling to deliver trusted data products in a secure, scalable fashion.

You will mentor engineers, contribute to platform infrastructure, and drive AI-assisted development practices, CI/CD, and observability with Grafana/Prometheus. The role emphasizes cost awareness and cross-team collaboration.