Data Engineer SR - Apache Flink

Encora Inc.; Belo Horizonte; Teletrabalho; BRL 80.000 - 120.000; Tempo integral

Resumo da oferta

A leading company is seeking a Data Engineer to design and maintain scalable data pipelines using Apache Flink and Kafka Connect. This role involves optimizing data processing for real-time insights and collaborating with product teams to meet data requirements. The ideal candidate will have strong experience with cloud platforms and data governance.

Qualificações

• Experience as a Data Engineer required.
• Strong experience with Apache Flink and Kafka Connect.
• Proficient in at least one programming language (Java or Python).

Responsabilidades

• Design and optimize data pipelines using Apache Flink.
• Implement and maintain Kafka Connect connectors.
• Build and maintain data infrastructure on Google Cloud Platform.

Conhecimentos

Data Engineering

Apache Flink

Kafka

SQL

Docker

Kubernetes

Python

Java

Git

Ferramentas

Google Cloud Platform

BigQuery

Apache Airflow

Tableau

Power BI

Prometheus

Grafana

Descrição da oferta de emprego

Important Information

Location:
Brazil

JobMode:
Full-time

Work Mode:
Work from home

Job Summary

As a Data Engineer, you will work with stream processing technologies, particularly Apache Flink and Kafka Connect. In this role, you'll design, implement, and maintain scalable data pipelines that process real-time and batch data, enabling critical business insights and data-driven decision making.

Responsibilities and Duties

1. Design, develop, and optimize data pipelines using Apache Flink for stream and batch processing;
2. Implement and maintain Kafka Connect connectors for seamless data integration;
3. Build and maintain data infrastructure on Google Cloud Platform (GCP);
4. Design and optimize BigQuery tables, views, and stored procedures;
5. Collaborate with product managers and analysts to understand data requirements;
6. Ensure data quality, reliability, and proper governance;
7. Troubleshoot and resolve data pipeline issues;
8. Document data flows, transformations, and architecture.

Essential Skills

• Experience as a Data Engineer;
• Strong experience with Apache Flink, including knowledge of the DataStream and Table APIs;
• Experience with Kafka and Kafka Connect for data integration;
• Experience with containerization (Docker) and orchestration (Kubernetes);
• Experience in Cloud Infrastructure platforms such as AWS, GCP;
• Solid SQL skills with specific experience in BigQuery SQL dialect;
• Experience with data modeling and schema design;
• Proficient in at least one programming language (Java or Python);
• Experience with version control systems (Git) and CI/CD pipelines.

Additional Skills

• Knowledge and hands-on experience with Kafka Infrastructure (i.e., managing and configuring Kafka components);
• Experience with other stream processing frameworks (Spark Streaming, Kafka Streams);
• Knowledge of data governance and security best practices;
• Proficiency with Google Cloud Platform services, particularly:

• Google Cloud Storage;
• Dataflow;
• Pub/Sub;
• BigQuery;

• Experience with Apache Airflow;
• Experience with BI tools such as Tableau or Power BI;
• Familiarity with monitoring tools like Prometheus and Grafana;
• Certifications such as Google Cloud Professional Data Engineer.

About Encora

Encora is the preferred digital engineering and modernization partner of some of the world’s leading enterprises and digital native companies. With over 9,000 experts in 47+ offices and innovation labs worldwide, Encora’s technology practices include Product Engineering & Development, Cloud Services, Quality Engineering, DevSecOps, Data & Analytics, Digital Experience, Cybersecurity, and AI & LLM Engineering.

At Encora, we hire professionals based solely on their skills and qualifications, and do not discriminate based on age, disability, religion, gender, sexual orientation, socioeconomic status, or nationality.