Senior Data Engineering Lead: Databricks & Spark on AWS

hackajob; Jersey City (NJ); On-site; USD 180,000 - 240,000; Full time

Benefits offered by this job

Health Care Coverage

On-Site Health and Wellness Centers

Retirement Savings Plan

Backup Childcare

Tuition Reimbursement

Mental Health Support

Financial Coaching

Job summary

hackajob is seeking a senior data/software engineer to lead the architecture and delivery of high-throughput, low-latency data pipelines using Databricks and Apache Spark in Jersey City, with deep hands-on involvement and strategic guidance.

You will oversee the setup and optimization of Databricks clusters, ensure policy compliance, and collaborate with cross-functional teams to design scalable data solutions on AWS, employing Python/Java and modern CI/CD practices.

Qualifications

• Formal training or certification in software engineering concepts.
• At least 10 years of professional software/data engineering experience.
• Strong proficiency in Python and/or Java.
• Hands-on expertise in Databricks and AWS services.

Responsibilities

• Lead architecture and delivery of high-throughput, low-latency data pipelines using Databricks and Apache Spark.
• Oversee the setup and optimization of Databricks clusters and ensure policy compliance.

Skills

Databricks

Apache Spark

AWS

Python

Java

ETL

Data Engineering

Cloud Computing

Data Pipelines

Data Governance

CI/CD

Performance Tuning

Data Modeling

Security

Incident Response

Automation

Education

Formal training or certification in software engineering concepts

Tools

Databricks

Apache Spark

AWS

Job description

hackajob is seeking a senior data/software engineer to lead the architecture and delivery of high-throughput, low-latency data pipelines using Databricks and Apache Spark in Jersey City, with deep hands-on involvement and strategic guidance.

You will oversee the setup and optimization of Databricks clusters, ensure policy compliance, and collaborate with cross-functional teams to design scalable data solutions on AWS, employing Python/Java and modern CI/CD practices.