Lead Data Engineer – Java, Spark & AWS

Next Frontier Capital; Jersey City (NJ); On-site; USD 150,000 - 230,000; Full time

Job summary

JPMorgan Chase & Co. in Jersey City, NJ invites a Lead Software Engineer to guide a data engineering team and build scalable ETL pipelines on Databricks. You will influence architecture and drive best-practice adoption across cloud services and data platforms.

You will lead development in Java or Python, design robust data pipelines, and mentor engineers while delivering high-impact financial data solutions in a complex enterprise environment.

Qualifications

• Formal training or certification in software engineering.
• 5+ years of hands-on programming in Java or Python for data engineering.
• Experience with Databricks and AWS data services including EMR, Athena, Glue, S3.
• Experience with Apache Spark for large-scale data processing and data pipelines.
• Experience leveraging AI tools for developer productivity.
• Experience with monitoring and logging tools such as Dynatrace, Splunk, Grafana, Prometheus.

Responsibilities

• Design, build, maintain and optimize robust ETL data pipelines using Databricks and ensure efficient ingestion, transformation and storage.
• Collaborate with data product owner, business stakeholders, and ensure best practices in data engineering, software engineering and resilient cloud architecture.
• Architect and implement data lake and data warehouse solutions leveraging AWS services (S3, Glue, SQS, SNS, Lambda, EMR).
• Collaborate with cross team to propose and build new solution for supporting overall application platform by the means of observability, orchestration, resiliency, developer experience, automation.
• Identifies opportunities to eliminate or automate remediation of recurring issues to improve overall operational stability of software applications and systems
• Lead and mentor a team of engineers, providing technical guidance and code reviews
• Monitor, troubleshoot, and tune data pipelines and cloud resources for optimal performance, reliability, and cost efficiency
• Promote reusability across data pipelines and operational simplicity by introducing good observability standards

Skills

Databricks (Spark, Delta Lake)

Java

Python

AWS data services

Apache Spark

CI/CD

DevOps

Infrastructure as Code

Monitoring/Logging

Mentoring/Leadership

Tools

SpringBoot

Docker

Kubernetes

Databricks

Spark

Java

Python

CI/CD

Infrastructure as Code

Job description

JPMorgan Chase & Co. in Jersey City, NJ invites a Lead Software Engineer to guide a data engineering team and build scalable ETL pipelines on Databricks. You will influence architecture and drive best-practice adoption across cloud services and data platforms.

You will lead development in Java or Python, design robust data pipelines, and mentor engineers while delivering high-impact financial data solutions in a complex enterprise environment.