Lead Big Data Engineer: Spark, ETL Pipelines & Cloud

Socket.dev; Plano (TX); On-site; USD 140,000 - 210,000; Full time

Job summary

JPMorganChase, within Corporate Technology, is seeking a Lead Software Engineer to design, automate, and operate scalable ETL/data transformation pipelines. You will mentor engineers and contribute to platform infrastructure, with emphasis on Spark, HBase, Cassandra, and cloud data tooling.

You will collaborate with stakeholders to gather requirements, implement batch/real-time workflows, and ensure secure, reliable, cost-efficient operations using CI/CD, Grafana/Prometheus, and API delivery via

Qualifications

• Formal training or certification on Software engineering concepts and 5+ years applied experience.
• Advanced in one or more programming languages and framework(s): Python, Java, Scala, Apache Spark, Databricks, Grafana, Prometheus, Elasticsearch, CloudWatch, Spring Boot API, and containers.
• Experience building scalable ETL/data pipelines and managing data lake tables.
• Experience with AI-assisted development tools and secure coding practices.
• Ability to drive CI/CD, monitoring, and cloud data tooling in production environments.

Responsibilities

• Design, implement, and optimize scalable ETL pipelines for batch and real-time data.
• Automate data transformations and monitor production data platforms.
• Collaborate with stakeholders to gather requirements for data workflows.
• Mentor junior engineers and promote secure coding and testing standards.
• Lead evaluation sessions with vendors and internal teams on architecture.
• Contribute to SDKs and infrastructure for data pipelines.

Skills

Python

Java

Scala

Apache Spark

Databricks

Grafana

Prometheus

Elasticsearch

CloudWatch

Spring Boot API

Docker

CI/CD

AI-assisted development

Tools

HBase

Cassandra

GitHub

Job description

JPMorganChase, within Corporate Technology, is seeking a Lead Software Engineer to design, automate, and operate scalable ETL/data transformation pipelines. You will mentor engineers and contribute to platform infrastructure, with emphasis on Spark, HBase, Cassandra, and cloud data tooling.

You will collaborate with stakeholders to gather requirements, implement batch/real-time workflows, and ensure secure, reliable, cost-efficient operations using CI/CD, Grafana/Prometheus, and API delivery via