Java Spark Engineer

Veriipro; Berkeley Heights (NJ); On-site; USD 140,000 - 190,000; Full time

Job summary

Veriipro in Berkeley Heights, NJ, seeks a senior data engineer to architect and implement scalable data pipelines using Spark (Java). You will lead batch and streaming ETL/ELT, optimize performance, and set coding standards across the team.

You will mentor engineers, drive data architecture decisions, partner with product and analytics, own production reliability, and help control cost and capacity for cluster infrastructure.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.
• 7+ years of professional Java development experience.
• 5+ years hands-on experience with Apache Spark in production environments.
• Expert-level understanding of distributed systems: fault tolerance, data locality, shuffle mechanics, resource management.
• Proven track record designing systems processing terabyte+ scale data.
• Strong SQL skills and deep familiarity with columnar storage formats (Parquet, ORC, Avro, Delta Lake/Iceberg).
• Experience with cluster managers (YARN, Kubernetes) and cloud-managed Spark.
• Proficiency with Kafka
• Strong grasp of CI/CD, containerization, and infrastructure-as-code practices.

Responsibilities

• Architect and build scalable, fault-tolerant data pipelines using Apache Spark (Java)
• Lead design of batch and streaming ETL/ELT systems handling large data volumes
• Deep-dive performance tuning: partitioning strategy, memory management, shuffle/skew optimization, job cost reduction
• Set coding standards and lead code/design reviews across the team
• Drive technical decisions on data architecture, storage formats, and pipeline orchestration
• Mentor mid-level and junior engineers; act as a technical escalation point
• Partner with product, analytics, and platform teams to translate requirements into scalable systems
• Own production reliability — on-call ownership, incident response, root‑cause analysis for pipeline failures
• Evaluate and introduce new tools/frameworks where they improve the system
• Contribute to capacity planning and cost optimization for cluster infrastructure

Skills

Java

Spark

Distributed systems

SQL

Kafka

CI/CD

Containers

Kubernetes

YARN

Delta Lake

Education

Bachelor’s or Master’s degree in CS/Engineering

Tools

YARN

Kubernetes

Docker

Job description

Primary Responsibilities

• Architect and build scalable, fault-tolerant data pipelines using Apache Spark (Java)
• Lead design of batch and streaming ETL/ELT systems handling large data volumes
• Deep-dive performance tuning: partitioning strategy, memory management, shuffle/skew optimization, job cost reduction
• Set coding standards and lead code/design reviews across the team
• Drive technical decisions on data architecture, storage formats, and pipeline orchestration
• Mentor mid-level and junior engineers; act as a technical escalation point
• Partner with product, analytics, and platform teams to translate requirements into scalable systems
• Own production reliability — on-call ownership, incident response, root‑cause analysis for pipeline failures
• Evaluate and introduce new tools/frameworks where they improve the system
• Contribute to capacity planning and cost optimization for cluster infrastructure

Required Qualifications

• Bachelor’s or Master’s degree in Computer Science, Engineering, or related field
• 7+ years of professional Java development experience
• 5+ years hands‑on experience with Apache Spark in production environments
• Expert‑level understanding of distributed systems: fault tolerance, data locality, shuffle mechanics, resource management
• Proven track record designing systems processing terabyte+ scale data
• Strong SQL skills and deep familiarity with columnar storage formats (Parquet, ORC, Avro, Delta Lake/Iceberg)
• Experience with cluster managers (YARN, Kubernetes) and cloud‑managed Spark
• Proficiency with Kafka
• Strong grasp of CI/CD, containerization, and infrastructure‑as‑code practices

Preferred Qualifications

• Experience with Flink or other stream‑processing frameworks
• Familiarity with data governance, lineage, and quality frameworks
• Experience with workflow orchestration at scale
• Background in system design for multi‑tenant or multi‑region data platforms
• Prior experience leading a team or acting as a technical lead