Senior Java Spark Architect — Data Pipelines Lead

Delta System & Software, Inc.; Berkeley Heights (NJ); On-site; USD 150,000 - 210,000; Full time

Job summary

Delta System & Software, Inc. is seeking a senior data engineer to architect and build scalable, fault-tolerant data pipelines using Java and Apache Spark. You will lead design of batch and streaming ETL/ELT systems, optimize performance, and set coding standards.

You will mentor engineers, own production reliability, partner with product and analytics teams, and drive decisions on data architecture, storage formats, and cluster orchestration.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.
• 7+ years of professional Java development experience.
• 5+ years hands-on experience with Apache Spark in production environments.
• Expert-level understanding of distributed systems: fault tolerance, data locality, shuffle mechanics, resource management.
• Proven track record designing systems processing terabyte+ scale data.
• Strong SQL skills and familiarity with columnar storage formats.

Responsibilities

• Architect and build scalable, fault-tolerant data pipelines using Java and Apache Spark.
• Lead design of batch and streaming ETL/ELT systems handling large data volumes.
• Deep-dive performance tuning: partitioning strategy, memory management, shuffle/skew optimization, job cost reduction.
• Set coding standards and lead code/design reviews across the team.
• Drive technical decisions on data architecture, storage formats, and pipeline orchestration.
• Mentor mid-level and junior engineers; act as a technical escalation point.
• Partner with product, analytics, and platform teams to translate requirements into scalable systems.
• Own production reliability — on-call ownership, incident response, root-cause analysis for pipeline failures.
• Evaluate and introduce new tools/frameworks where they improve the system.
• Contribute to capacity planning and cost optimization for cluster infrastructure.

Skills

Java

Apache Spark

Distributed systems

SQL

CI/CD

Kubernetes

Performance tuning

Kafka

Education

Bachelor’s or Master’s degree in Computer Science, Engineering, or related field

Tools

YARN

Kubernetes

Delta Lake

Parquet/ORC/Avro

Kafka

Job description

Delta System & Software, Inc. is seeking a senior data engineer to architect and build scalable, fault-tolerant data pipelines using Java and Apache Spark. You will lead design of batch and streaming ETL/ELT systems, optimize performance, and set coding standards.

You will mentor engineers, own production reliability, partner with product and analytics teams, and drive decisions on data architecture, storage formats, and cluster orchestration.