Lead Software Engineer - Databricks, Spark, AWS

hackajob; Jersey City (NJ); On-site; USD 180,000 - 240,000; Full time

Benefits offered by this job

Health Care Coverage

On-Site Health and Wellness Centers

Retirement Savings Plan

Backup Childcare

Tuition Reimbursement

Mental Health Support

Financial Coaching

Job summary

hackajob is seeking a senior data/software engineer to lead the architecture and delivery of high-throughput, low-latency data pipelines using Databricks and Apache Spark in Jersey City, with deep hands-on involvement and strategic guidance.

You will oversee the setup and optimization of Databricks clusters, ensure policy compliance, and collaborate with cross-functional teams to design scalable data solutions on AWS, employing Python/Java and modern CI/CD practices.

Qualifications

• Formal training or certification in software engineering concepts.
• At least 10 years of professional software/data engineering experience.
• Strong proficiency in Python and/or Java.
• Hands-on expertise in Databricks and AWS services.

Responsibilities

• Lead architecture and delivery of high-throughput, low-latency data pipelines using Databricks and Apache Spark.
• Oversee the setup and optimization of Databricks clusters and ensure policy compliance.

Skills

Databricks

Apache Spark

AWS

Python

Java

ETL

Data Engineering

Cloud Computing

Data Pipelines

Data Governance

CI/CD

Performance Tuning

Data Modeling

Security

Incident Response

Automation

Education

Formal training or certification in software engineering concepts

Tools

Databricks

Apache Spark

AWS

Job description

Lead the architecture and delivery of high-throughput, low-latency data pipelines using Databricks and Apache Spark. Oversee the setup and optimization of Databricks clusters and ensure compliance with organizational policies.

Requirements

Candidates should have formal training or certification in software engineering concepts and at least 10 years of professional software/data engineering experience. Strong proficiency in Python and/or Java, along with hands-on expertise in Databricks and AWS services, is required.

Key Skills

• Databricks
• Apache Spark
• AWS
• Python
• Java
• ETL
• Data Engineering
• Cloud Computing
• Data Pipelines
• Data Governance
• CI/CD
• Performance Tuning
• Data Modeling
• Security
• Incident Response
• Automation

Benefits

• Health Care Coverage
• On-Site Health and Wellness Centers
• Retirement Savings Plan
• Backup Childcare
• Tuition Reimbursement
• Mental Health Support
• Financial Coaching