Data Engineer (Cloudera)

Adentis Portugal; Lisboa; Híbrido; EUR 50 000 - 70 000; Tempo integral

Vantagens oferecidas por esta oferta de emprego

Health benefits

Flexible schedule

Training & Certification

Career progression

Work-Life balance

Resumo da oferta

ADENTIS Portugal is seeking a data engineer to design and optimize distributed data pipelines using Apache Spark within a large-scale environment. You will focus on performance, caching strategies, and SQL tuning to handle big data workloads.

The role involves collaborating with cross-functional teams, working on Cloudera-based platforms, and gradually taking on DevOps responsibilities to become autonomous in managing data workflows.

Qualificações

• Expertise in Apache Spark and distributed computation techniques.
• Experience with advanced SQL optimization and caching strategies.
• Knowledge of the Cloudera ecosystem for big data platforms.

Responsabilidades

• Develop and optimize distributed data pipelines using Apache Spark.
• Implement caching techniques to improve data retrieval times.
• Optimize partition keys joins to speed up query processing.
• Tune SQL queries for performance in large-scale systems.
• Leverage Cloudera platform to manage data workflows.
• Learn DevOps duties (5-10% of time) to gain autonomy.

Conhecimentos

Apache Spark

Distributed computation

SQL optimization

Ferramentas

Cloudera ecosystem

Containerization

Orchestration

Automation & Monitoring

Descrição da oferta de emprego

With just over 9 years of experience in the Portuguese market, we share our DNA with more than 200 workers and position our offer according to 3 lines of service:

• Strategy (Outsourcing, NeXel, Team as a Service, Tech Academies);
• Nearshore.

In ADENTIS, we focus on PEOPLE. This is our emotional salary:

• Great Work-Life balance;
• Very flexible organizational routine;
• Health benefits (for you and your family);
• Over 300 protocols to offer you great discounts in different areas;
• Continuous professional development sponsored by our Training and Certification Department;
• Regular feedback on your performance through a personalized plan;
• Comprehensive career plan and progression involving assertive performance reviews.

Responsibilities:

• Distributed Data Processing: Develop and optimize distributed data pipelines using Apache Spark, ensuring efficient processing of large datasets
• Caching & Computation Optimization: Implement advanced caching techniques and optimize distributed computation workflows to improve data retrieval times and resource utilization
• Partition Key Join Optimization: Optimize partition keys joins to speed up query processing and enhance overall system performance in distributed environments
• SQL Query Optimization: Optimize SQL queries for better performance and scalability, particularly in large-scale, distributed database systems
• Cloudera Ecosystem: If applicable, leverage the Cloudera platform to manage and scale data workflows in big data ecosystems, ensuring reliability and efficiency
• Wants to learn more about DevOps to be autonomous, with the following DevOps responsibilities (5-10% of the time)

Requirements:

• Expertise in Apache Spark and distributed computation techniques
• Experience with advanced SQL optimization and caching strategies
• Knowledge of the Cloudera ecosystem is a valuable asset for managing big data platforms
• Ability to work independently and take initiative to solve technical problems
• Excellent communication skills to effectively collaborate with technical and non-technical teams.
• Version Control & Branching
• Containerization & Orchestration
• Automation & Monitoring
• Available to go to the office twice per week