Software Engineer (Apache Hudi) - Remote Work | REF#298103

BairesDev; United States; Remote; USD 120,000 - 160,000; Full time

Benefits offered by this job

Remote work (100% from anywhere)

USD compensation or local currency

Home office hardware & software setup

Flexible hours

Parental leaves & vacations

Global, multicultural team

Mentorship & growth opportunities

Job summary

BairesDev is seeking a Software Engineer specializing in Apache Hudi to design and manage large-scale incremental data processing on data lakes. You will develop efficient pipelines handling upserts, deletes, and CDC, while optimizing storage formats to boost query performance and reduce latency.

The role emphasizes ACID transactions, schema evolution, and collaboration with data teams to integrate Hudi with streaming and batch engines. Excellent English skills required.

Qualifications

• 4+ years of software or data engineering experience.
• Proven expertise with Apache Hudi in incremental data processing.
• Experience with upserts, deletes, and streaming ingestion into data lake storage.
• Strong understanding of data lakehouse architectures and distributed systems.
• Advanced English proficiency.

Responsibilities

• Design, implement, and manage Apache Hudi tables for large-scale incremental processing.
• Build data pipelines handling high-frequency upserts, deletes, and CDC.
• Optimize storage formats and indexing to reduce data-lake latency.
• Implement ACID transactions and schema evolution for consistency.
• Collaborate with data teams to integrate Hudi with streaming and batch engines.
• Monitor and tune table services like compaction and clustering.

Skills

Data engineering

Distributed systems

English proficiency

Tools

Apache Hudi

Data lakehouse

CDC

Job description

At BairesDev®, we've been leading the way in technology projects for over 15 years. We deliver cutting-edge solutions to giants like Google and the most innovative startups in Silicon Valley.

Our diverse 4,000+ team, composed of the world's Top 1% of tech talent, works remotely on roles that drive significant impact worldwide.

When you apply for this position, you're taking the first step in a process that goes beyond the ordinary. We aim to align your passions and skills with our vacancies, setting you on a path to exceptional career development and success.

As a Software Engineer specializing in Apache Hudi, you will work with incremental data processing on data lakes to ensure high performance and data integrity. You will be responsible for managing complex data ingestion and storage patterns, providing development teams with a scalable and reliable foundation for near real-time analytics.

What You'll Do

• Design, implement, and manage Apache Hudi tables to support large-scale incremental data processing.
• Develop efficient data pipelines that handle high-frequency upserts, deletes, and change data capture (CDC).
• Optimize storage formats and indexing strategies to improve query performance and reduce latency on the data lake.
• Implement ACID transactions and schema evolution to maintain data consistency across distributed systems.
• Collaborate with data engineering teams to integrate Hudi with streaming and batch processing engines.
• Monitor and tune background table services, including compaction and clustering, to ensure optimal resource utilization.

What We Are Looking For

• 4+ years of experience in Software Engineering or Data Engineering.
• Proven expertise in building and optimizing incremental data processing pipelines using Apache Hudi.
• Hands-on experience managing upserts, deletes, and streaming ingestion into data lake storage.
• Deep understanding of data lakehouse architectures and distributed systems.
• Advanced proficiency in English.

How we do make your work (and your life) easier:

• 100% remote work (from anywhere).
• Excellent compensation in USD or your local currency if preferred.
• Hardware and software setup for you to work from home.
• Flexible hours: create your own schedule.
• Paid parental leaves, vacations, and national holidays.
• Innovative and multicultural work environment: collaborate and learn from the global Top 1% of talent.
• Supportive environment with mentorship, promotions, skill development, and diverse growth opportunities.

Join a global team where your unique talents can truly thrive and make a significant impact!