Senior Data Engineer

Jobgether; Brasil; Remote; BRL 274,000 - 438,000; Full time

Benefits offered by this job

Remote work

Equity opportunities

Flexible office memberships

Market-competitive salary

International collaboration

Job summary

Jobgether in Brazil is seeking a Senior Data Engineer to own a next-generation data platform built around real-time streaming and lakehouse technologies. You will design, develop, and operate scalable data pipelines translating operational data into reliable data products, working with Spark Structured Streaming, Iceberg, CDC, and AWS infrastructure.

You will also oversee compute resources, optimize performance and costs, implement monitoring and observability, and help teams adopt self-service

Qualifications

• At least 7 years of professional experience in data engineering.

Responsibilities

• Design, build, and maintain Spark Structured Streaming pipelines processing CDC events into near-real-time data models.
• Implement reliable stateful processing, SCDs, checkpointing, and safe recovery mechanisms.
• Manage AWS EMR clusters and optimize resource allocation and costs.
• Maintain and optimize Iceberg tables, including partitioning, compaction, and snapshots.
• Onboard new domains by configuring CDC sources, Kafka topics, and validating backfills in Snowflake.
• Troubleshoot complex production incidents and perform root-cause analysis.
• Develop monitoring and observability with Grafana Loki and related tooling.
• Improve CI/CD, deployment workflows, and self-service data platform capabilities.
• Collaborate with partner engineering teams and cross-functional stakeholders.

Skills

Data engineering

Streaming pipelines

Problem solving

Communication

Tools

Apache Spark

Spark Structured Streaming

Iceberg

Kafka

Airflow

Kubernetes

Snowflake

Estuary Flow

Grafana Loki

Job description

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Senior Data Engineer based in Brazil.

We are looking for an experienced Senior Data Engineer to take ownership of a next-generation data platform built around real-time streaming and modern lakehouse technologies. In this role, you will design, develop, and operate scalable data pipelines that transform operational data into reliable, business-critical data products. You will work extensively with Apache Spark Structured Streaming, Change Data Capture (CDC), Apache Iceberg, and AWS infrastructure to deliver data transformations in near real time. Beyond pipeline development, you will oversee compute infrastructure, optimize performance and costs, and ensure the reliability of production systems. You will also build observability tools and improve developer workflows, enabling engineering teams to onboard and manage their own data models. This is an opportunity to influence the evolution of a large-scale data ecosystem while solving complex technical challenges in a distributed, international environment.

Accountabilities:

• Streaming Data Pipeline Development: Design, build, and maintain robust Apache Spark Structured Streaming applications that process Change Data Capture (CDC) events into reliable, near-real-time data models.
• Stream Processing Reliability: Implement and optimize stateful processing, slowly changing dimensions (SCD), checkpointing, state stores, late-arriving and partial records, and safe recovery mechanisms to ensure accurate and consistent data processing.
• AWS Infrastructure Management: Own and scale AWS EMR clusters, managing instance sizing, resource allocation, managed scaling, multi-cluster deployments, driver and executor memory optimization, and workload consolidation to improve efficiency and reduce infrastructure costs.
• Lakehouse Management: Maintain and optimize Apache Iceberg tables at scale by designing partitioning strategies and implementing compaction, snapshot expiration, and delete-file cleanup, including specialized maintenance for high-churn tables.
• Data Source Integration: Onboard new business domains onto the streaming platform by configuring CDC sources and destinations through Estuary Flow, setting up Kafka-compatible topics, validating backfills in Snowflake, and coordinating with source database owners.
• Production Troubleshooting: Investigate and resolve complex production incidents across the data platform, addressing issues such as connection failures, offset regressions, token expiration, schema inconsistencies, and write conflicts while focusing on root-cause resolution.
• Monitoring and Observability: Develop monitoring capabilities and diagnostic tools to improve incident detection and resolution, including integrating Spark driver and executor logs with Grafana Loki, implementing watchdog mechanisms, and creating AI-assisted operational runbooks.
• CI/CD and Developer Experience: Improve deployment workflows, configuration management, automated pipelines, and operational safeguards to enable engineering teams to independently and safely deploy new data models.
• Cross-Team Collaboration: Support partner engineering teams with ad hoc data ingestion, backfill requests, technical escalations, and platform-related guidance.

Requirements:

• At least 7 years of professional experience in data engineering, including a minimum of 3 years operating production-grade streaming data pipelines.
• Apache Spark Expertise: Strong hands-on experience with Apache Spark, particularly Structured Streaming, state stores such as RocksDB, checkpointing, stateful processing, and performance optimization.
• Lakehouse Technologies: Production experience with modern lakehouse table formats, ideally Apache Iceberg, including partitioning strategies, table maintenance, compaction, and snapshot management. Experience with Delta Lake or Apache Hudi is also relevant.
• Cloud Infrastructure: Experience operating Spark workloads on AWS EMR, Amazon EKS, or comparable platforms, with a strong understanding of cluster management, capacity planning, autoscaling, resource allocation, and infrastructure cost optimization.
• Streaming and CDC: Solid understanding of Change Data Capture principles and Kafka semantics, including offsets, consumer configurations, message delivery guarantees, and event processing reliability.
• Programming Skills: Strong proficiency in Python and SQL. Experience with Scala or Java is considered an advantage.
• Orchestration and Automation: Experience with Apache Airflow or similar workflow orchestration tools, alongside CI/CD pipelines and infrastructure-as-code practices.
• Troubleshooting and Analytical Skills: A systematic approach to debugging, with the ability to interpret logs, investigate complex issues, formulate hypotheses, and validate solutions using data.
• Communication and Collaboration: Ability to work independently while collaborating effectively with distributed engineering teams and cross-functional stakeholders.
• Nice to Have: Experience with Snowflake, particularly querying external or Iceberg tables, as well as monitoring and observability tools such as Grafana, Loki, or Alloy.
• Platform Enablement: Previous experience building self-service data platforms or enabling other engineering teams to independently manage data workflows is an advantage.
• Containerization: Familiarity with Kubernetes (K8s) is a plus.

Benefits:

• Competitive Compensation: Annual gross base salary ranging from $50,000 to $80,000 USD, depending on experience, technical skills, and location.
• Localized Compensation: Salary offers are adjusted to the candidate's country and paid in the relevant local currency using market-aligned conversion rates.
• Equity Opportunities: Stock grant opportunities, depending on role, employment status, and location.
• Flexible Working Arrangements: Fully remote work, with optional flexible office memberships through IWG.
• Additional Benefits: Access to additional perks and benefits based on employment status and country of residence.
• International Exposure: Opportunity to collaborate with globally distributed engineering teams and contribute to a large-scale international technology platform.
• Technical Growth: Work with modern data engineering technologies, including Spark Structured Streaming, Apache Iceberg, AWS, CDC, and AI-assisted operational tools.
• Meaningful Technical Ownership: Take responsibility for the architecture, reliability, scalability, and continuous improvement of a business-critical data platform.
• Inclusive Workplace: Join an environment committed to equal opportunities, diversity, accessibility, and professional development.