Senior Java Spark Architect & Data Pipeline Tech Lead

Mphasis; New York (NY); On-site; USD 63,000 - 135,000; Full time

Job summary

Mphasis seeks an experienced Java Spark Engineer to design and operate scalable data pipelines in a large-scale environment. You will drive batch and streaming ETL/ELT systems across multi-tenant data platforms, ensuring fault tolerance and optimal performance.

You will mentor engineers, define coding standards, and collaborate with product, analytics, and platform teams to deliver reliable, cost-efficient data solutions in a 5-day in-office setting in New Jersey.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.
• 7+ years of professional Java development experience.
• 5+ years hands-on Apache Spark in production environments.
• Expert-level understanding of distributed systems: fault tolerance, data locality, shuffle mechanics, resource management.
• Proven track record designing systems processing terabyte+ scale data.
• Strong SQL skills and deep familiarity with columnar storage formats (Parquet, ORC, Avro, Delta Lake/Iceberg).
• Experience with cluster managers (YARN, Kubernetes) and cloud-managed Spark.
• Proficiency with Kafka.
• Strong grasp of CI/CD, containerization, and infrastructure-as-code practices.

Responsibilities

• Architect and build scalable, fault-tolerant data pipelines using Apache Spark (Java).
• Lead design of batch and streaming ETL/ELT systems handling large data volumes.
• Deep-dive performance tuning: partitioning, memory management, shuffle/skew optimization, cost reduction.
• Set coding standards and lead code/design reviews across the team.
• Drive decisions on data architecture, storage formats, and pipeline orchestration.
• Mentor mid-level and junior engineers; act as a technical escalation point.
• Partner with product, analytics, and platform teams to translate requirements into scalable systems.
• Own production reliability — on-call, incident response, root-cause analysis for pipeline failures.
• Evaluate and introduce new tools/frameworks to improve the system.
• Contribute to capacity planning and cost optimization for cluster infrastructure.

Skills

Java

Apache Spark

Distributed systems

SQL

Kubernetes

Kafka

Education

Bachelor’s or Master’s degree in Computer Science or related field

Tools

Hadoop YARN

Kubernetes

Delta Lake

Parquet/ORC/Avro

CI/CD

Job description

Mphasis seeks an experienced Java Spark Engineer to design and operate scalable data pipelines in a large-scale environment. You will drive batch and streaming ETL/ELT systems across multi-tenant data platforms, ensuring fault tolerance and optimal performance.

You will mentor engineers, define coding standards, and collaborate with product, analytics, and platform teams to deliver reliable, cost-efficient data solutions in a 5-day in-office setting in New Jersey.