Data Engineer: Spark & Cloudera Big Data Focus

Adentis Portugal; Lisboa; Híbrido; EUR 50 000 - 70 000; Tempo integral

Vantagens oferecidas por esta oferta de emprego

Health benefits

Flexible schedule

Training & Certification

Career progression

Work-Life balance

Resumo da oferta

ADENTIS Portugal is seeking a data engineer to design and optimize distributed data pipelines using Apache Spark within a large-scale environment. You will focus on performance, caching strategies, and SQL tuning to handle big data workloads.

The role involves collaborating with cross-functional teams, working on Cloudera-based platforms, and gradually taking on DevOps responsibilities to become autonomous in managing data workflows.

Qualificações

• Expertise in Apache Spark and distributed computation techniques.
• Experience with advanced SQL optimization and caching strategies.
• Knowledge of the Cloudera ecosystem for big data platforms.

Responsabilidades

• Develop and optimize distributed data pipelines using Apache Spark.
• Implement caching techniques to improve data retrieval times.
• Optimize partition keys joins to speed up query processing.
• Tune SQL queries for performance in large-scale systems.
• Leverage Cloudera platform to manage data workflows.
• Learn DevOps duties (5-10% of time) to gain autonomy.

Conhecimentos

Apache Spark

Distributed computation

SQL optimization

Ferramentas

Cloudera ecosystem

Containerization

Orchestration

Automation & Monitoring

Descrição da oferta de emprego

ADENTIS Portugal is seeking a data engineer to design and optimize distributed data pipelines using Apache Spark within a large-scale environment. You will focus on performance, caching strategies, and SQL tuning to handle big data workloads.

The role involves collaborating with cross-functional teams, working on Cloudera-based platforms, and gradually taking on DevOps responsibilities to become autonomous in managing data workflows.