Senior Data Engineer

Bebeedataengineer; Manaus; Presencial; BRL 80.000 - 120.000; Tempo integral

Resumo da oferta

A data engineering firm in Manaus, Brazil, is seeking a skilled Data Engineer to build scalable data pipelines and optimize ETL/ELT processes. The ideal candidate will have strong experience with Spark, PySpark, SQL, and Delta Lake, focusing on data ingestion and transformation. This role includes developing reusable frameworks and delivering insights through Power BI dashboards. Candidates must troubleshoot performance issues to ensure high data quality and reliability.

Qualificações

• Strong experience in building scalable data pipelines using Spark, PySpark, SQL, and Delta Lake.
• Experience with Databricks for ETL/ELT development.

Responsabilidades

• Design and optimize ETL / ELT pipelines in Databricks.
• Extract, clean, and transform data from various sources.
• Develop reusable pipeline frameworks and data validation logic.
• Deliver insights through Power BI dashboards.
• Implement best practices for lakehouse development.
• Troubleshoot pipeline performance issues.

Conhecimentos

Building scalable data pipelines

Spark

PySpark

SQL

Delta Lake

Data transformation

ETL/ELT design

Power BI

Descrição da oferta de emprego

Key Responsibilities

The ideal candidate will possess strong experience in building scalable data pipelines using Spark, PySpark, SQL, and Delta Lake. This role focuses on ingesting data from multiple sources, transforming it for analytics, and publishing high-quality datasets and visualizations.

• Design and optimize ETL / ELT pipelines in Databricks using PySpark, Spark SQL, and Delta Lake.
• Extract, clean, and transform data from diverse sources (APIs, SQL databases, cloud storage, SAP / legacy systems, streaming).
• Develop reusable pipeline frameworks, data validation logic, and performance-tuned transformations.
• Curate datasets and deliver insights through Power BI dashboards.
• Implement best practices for lakehouse development, orchestration, and version control.
• Troubleshoot pipeline performance issues and ensure data accuracy, reliability, and quality.