Senior Java Spark Architect — On-Site Data Pipelines Lead

Tech Mirrors; Cleburne (TX); On-site; USD 140,000 - 190,000; Full time

Job summary

Tech Mirrors is seeking a Java Spark Engineer to architect and build scalable data pipelines using Apache Spark in a flexible NJ-based environment. The role focuses on batch and streaming ETL/ELT systems, performance tuning, and setting coding standards for the team.

Ideal candidates bring 7+ years of Java and 5+ years of Spark production experience, with strong distributed systems knowledge, SQL proficiency, and containerization/CI/CD expertise.

Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, or related field.
• 7+ years of professional Java development experience.
• 5+ years hands-on experience with Apache Spark in production environments.
• Expert-level understanding of distributed systems: fault tolerance, data locality, shuffle mechanics, resource management.
• Proven track record designing systems processing terabyte+ scale data.
• Strong SQL skills and deep familiarity with columnar storage formats (Parquet, ORC, Avro, Delta Lake/ Iceberg).
• Experience with cluster managers (YARN, Kubernetes) and cloud-managed Spark.
• Proficiency with Kafka.
• Strong grasp of CI/CD, containerization, and infrastructure-as-code practices.

Responsibilities

• Architect and build scalable, fault-tolerant data pipelines using Apache Spark (Java)
• Lead design of batch and streaming ETL/ELT systems handling large data volumes
• Deep-dive performance tuning: partitioning strategy, memory management, shuffle/skew optimization
• Set coding standards and lead code/design reviews across the team
• Drive technical decisions on data architecture, storage formats, and pipeline orchestration
• Mentor mid-level and junior engineers; act as a technical escalation point
• Partner with product, analytics, and platform teams to translate requirements into scalable systems
• Own production reliability - on-call ownership, incident response, root-cause analysis for pipeline failures
• Evaluate and introduce new tools/frameworks where they improve the system
• Contribute to capacity planning and cost optimization for cluster infrastructure

Skills

Java development

Distributed systems

SQL

CI/CD

Containerization

Education

Bachelor's or Master's in CS/Engineering

Tools

Apache Spark

YARN

Kubernetes

Kafka

Parquet/ORC/Avro/Delta Lake

Job description

Tech Mirrors is seeking a Java Spark Engineer to architect and build scalable data pipelines using Apache Spark in a flexible NJ-based environment. The role focuses on batch and streaming ETL/ELT systems, performance tuning, and setting coding standards for the team.

Ideal candidates bring 7+ years of Java and 5+ years of Spark production experience, with strong distributed systems knowledge, SQL proficiency, and containerization/CI/CD expertise.