Distributed Systems Engineer (Apache Spark Internals) - Remote Work | REF#308566

BairesDev; Montenegro; Remote; BRL 657,000 - 986,000; Full time

Benefits offered by this job

Remote Work

Competitive USD compensation

Hardware and software setup for home工作

Flexible hours

Parental leaves, vacations and public

Multicultural, global Top 1% talent

Mentorship and growth opportunities

Job summary

BairesDev is seeking a Distributed Systems Engineer with Apache Spark internals experience to work on the engine's core components at petabyte scale. You will tackle Catalyst, Tungsten, DAG Scheduler, Shuffle, and memory management, contributing to Spark's engine-level code and collaborating with open source committers.

Ideal candidates have 6+ years in software development, strong Java/Scala skills, and a deep understanding of distributed systems and JVM performance.

Qualifications

• 6+ years in software development.
• Strong Java and/or Scala skills.
• Experience with distributed systems and concurrent or parallel programming.
• Working knowledge of Spark internals (Catalyst, Tungsten, DAG Scheduler, Shuffle, memory management).
• Familiarity with JVM performance characteristics (GC, memory, threading).
• Advanced level of English.

Responsibilities

• Contribute production-grade code to the Apache Spark project, particularly in Spark SQL and Structured Streaming.
• Debug and optimize Spark internals — Catalyst, Tungsten, DAG Scheduler, Shuffle, and memory management — at petabyte scale.
• Influence architectural direction for Spark performance and scalability.
• Profile and tune JVM behavior (GC, memory layout, concurrency) at the engine level.
• Collaborate with cross-functional engineering teams and open source committers on integrations and ecosystem work.
• Mentor senior engineers and raise the engineering bar through code reviews and design critiques.

Skills

Java/Scala

Distributed systems

Concurrency/Parallel programming

Spark internals

JVM performance

Tools

Apache Spark

Iceberg

Hive

Hadoop

Arrow

Parquet

Job description

At BairesDev® we've been leading the way in technology projects for over 15 years. We deliver cutting-edge solutions to giants like Google and the most innovative startups in Silicon Valley.

Our diverse 4,000+ team, composed of the world's Top 1% of tech talent, works remotely on roles that drive significant impact worldwide.

When you apply for this position, you're taking the first step in a process that goes beyond the ordinary. We aim to align your passions and skills with our vacancies, setting you on a path to exceptional career development and success.

Distributed Systems Engineer (Apache Spark Internals) at BairesDev

We are seeking a Distributed Systems Engineer with experience in Apache Spark internals to work on the engine itself, not pipelines built on top of it. The work covers the core components of Spark — Catalyst Optimizer, Tungsten execution, DAG Scheduler, Shuffle subsystem, and memory management — at petabyte scale. The technology stack centers on the open source Apache Spark project and adjacent ecosystem projects (Iceberg, Hive, Hadoop, Arrow, Parquet), with ample scope for upstream contribution. We're looking for engineers who have worked on engine-level code in Spark or comparable distributed compute systems and are comfortable debugging concurrency, memory, and execution issues at scale.

What You'll Do

• Contribute production-grade code to the Apache Spark project, particularly in Spark SQL and Structured Streaming.
• Debug and optimize Spark internals — Catalyst, Tungsten, DAG Scheduler, Shuffle, and memory management — at petabyte scale.
• Influence architectural direction for Spark performance and scalability.
• Profile and tune JVM behavior (GC, memory layout, concurrency) at the engine level.
• Collaborate with cross-functional engineering teams and open source committers on integrations and ecosystem work.
• Mentor senior engineers and raise the engineering bar through code reviews and design critiques.

What We Are Looking For

• 6+ years of experience in software development.
• Strong Java and/or Scala skills.
• Experience with distributed systems and concurrent or parallel programming.
• Working knowledge of Spark internals (Catalyst, Tungsten, DAG Scheduler, Shuffle, or memory management).
• Familiarity with JVM performance characteristics (GC, memory, threading).
• Advanced level of English.

Nice To Have

• Upstream contributions to Apache Spark (merged PRs in Spark SQL or Structured Streaming); committer status is a strong plus.
• Contributions to Apache Iceberg, Hive, Hadoop, Arrow, or Parquet.
• Kubernetes and cloud-native Spark deployment experience.
• Experience operating Spark at petabyte scale in production.

How we do make your work (and your life) easier:

• Remote Work.
• Excellent compensation in USD or your local currency if preferred.
• Hardware and software setup for you to work from home.
• Flexible hours: create your own schedule.
• Paid parental leaves, vacations, and national holidays.
• Innovative and multicultural work environment: collaborate and learn from the global Top 1% of talent.
• Supportive environment with mentorship, promotions, skill development, and diverse growth opportunities.

Join a global team where your unique talents can truly thrive and make a significant impact!