Big Data Developer | Spark & Scala

act digital; Lisboa; Presencial; EUR 90.000 - 120.000; Jornada completa

Descripción de la vacante

act digital seeks a Senior Big Data Engineer to design, develop, and optimize large-scale data solutions on a Cloudera Hadoop platform. You will work mainly with Apache Spark and Scala, shaping features, tuning workloads, and enhancing scalability while maintaining robust production-grade systems.

You will participate in architecture improvements, automated testing, CI/CD, security, and continuous improvement within an Agile team. Strong English (B2) is required.

Formación

• Strong professional experience in Big Data Engineering.
• Advanced knowledge of Apache Spark 3.
• Proven experience optimizing Spark applications.
• Strong development experience with Scala 2.
• Advanced knowledge of the Hadoop ecosystem (YARN, HDFS, Hive, Kafka).
• Experience with CI/CD pipelines and Agile/Scrum practices.

Responsabilidades

• Develop and evolve high-performance Big Data applications.
• Build and optimize distributed processing solutions using Spark and Scala.
• Improve architecture through component rationalization and reuse.
• Optimize Spark workloads for scalability, reliability, and performance.
• Work across the Hadoop ecosystem including HDFS, Hive, Kafka and YARN.
• Contribute to CI/CD, deployment, and production-readiness activities.

Conocimientos

Big Data Engineering

Apache Spark 3

Scala 2.12

Hadoop

CI/CD

Agile/Scrum

English B2

Educación

Bachelor's degree in Computer Science

Herramientas

Jenkins

XL Deploy

XL Release

Bitbucket

Jira

Confluence

Descripción del empleo

We are looking for an experienced Senior Big Data Engineer to join a team responsible for developing and evolving large-scale data solutions on an enterprise Cloudera Hadoop platform.

You will work primarily with Apache Spark and Scala, designing new features, optimizing distributed data processing workloads, and improving the scalability and maintainability of existing solutions.

Beyond development, you will contribute to architecture improvements, automated testing, CI/CD, deployments, security, and continuous improvement within a collaborative Agile environment.

As a Senior Big Data Engineer, you will:

• Develop and evolve high-performance Big Data applications.
• Build and optimize distributed processing solutions using Spark and Scala.
• Improve architecture through component rationalization and reuse.
• Optimize Spark workloads for scalability, reliability, and performance.
• Work across the Hadoop ecosystem, including HDFS, Hive, Kafka, and YARN.
• Contribute to CI/CD, deployment, and production-readiness activities.
• Promote code quality through reviews, testing, and engineering best practices.
• Design, develop, and maintain Big Data applications using Spark 3 and Scala 2.12.
• Develop new features according to technical and business requirements.
• Build reusable and scalable data processing components.
• Ensure solutions remain maintainable, consistent, and performance-oriented.
• Troubleshoot complex data processing issues.
• Contribute to the continuous evolution of existing applications.

Spark Performance & Optimization

• Design and optimize distributed Spark applications.
• Analyze and improve:
• Execution plans
• Joins
• Caching strategies
• Identify processing bottlenecks and implement sustainable optimizations.
• Ensure applications make efficient use of the underlying Hadoop infrastructure.
• Balance performance improvements with maintainability and operational stability.
• Develop robust and maintainable applications using Scala.
• Apply appropriate functional programming principles.
• Promote clean coding practices, testability, and component reuse.
• Participate in technical design and code reviews.
• Use frameworks such as ZIO or Cats where relevant.

Hadoop & Data Platform

• Develop and support solutions running on an on-premises Cloudera Hadoop ecosystem.
• Work with technologies including:
• HDFS
• YARN
• Hive
• HBase
• Kafka
• Understand interactions between data processing applications and platform components.
• Troubleshoot application and data-flow issues across the Hadoop ecosystem.

Kafka & Data Integration

• Work with Apache Kafka for distributed data exchange and streaming use cases.
• Support integrations between applications and data sources.
• Troubleshoot Kafka-related processing and connectivity issues.
• Contribute to reliable and scalable data flows.

Security

• Work within secured Big Data environments.
• Apply and troubleshoot security mechanisms including:
• Ensure solutions comply with platform access and security requirements.
• Collaborate with infrastructure teams when investigating authorization and authentication issues.

CI/CD & DevOps

• Develop and maintain CI/CD pipelines for Big Data applications.
• Support automated build, testing, packaging, and deployment activities.
• Participate in deployments across test and production environments.
• Work with:
• Jenkins
• XL Deploy
• XL Release
• Bitbucket
• Improve delivery reliability and reduce manual deployment activities.
• Manage component upgrades and associated validation activities.

Scheduling & Operations

• Schedule and monitor data processing workloads using Control-M.
• Support dependencies between scheduled jobs and data-processing workflows.
• Investigate failed executions and operational issues.
• Contribute to stable application operation across production environments.

Code Quality & Continuous Improvement

• Participate actively in peer code reviews.
• Promote coding standards and development best practices.
• Improve existing components through rationalization, refactoring, and reuse.
• Reduce unnecessary duplication and technical debt.
• Contribute to retrospectives and continuous improvement initiatives.
• Support technical upgrades while maintaining solution stability.
• Work within a Scrum environment.
• Sprint Planning
• Daily Meetings
• Sprint Reviews
• Use Jira and Confluence for delivery and documentation.
• Estimate and plan technical activities.
• Communicate effectively with Infrastructure Engineers, Developers, Business Analysts, and end users.

What We're Looking For

Mandatory Requirements

• Strong professional experience in Big Data Engineering.
• Advanced knowledge of Apache Spark 3.
• Proven experience optimizing Spark applications.
• Strong development experience with Scala 2.
• Advanced knowledge of the Hadoop ecosystem, particularly:
• YARN
• HDFS
• Hive
• Kafka
• Experience developing maintainable and reusable software components.
• Strong understanding of coding best practices.
• Knowledge of Big Data security concepts, including:
• Experience implementing or working with CI/CD pipelines.
• Comfortable developing and troubleshooting complex distributed applications.
• Experience working within Agile/Scrum teams.
• English B2 minimum, with C1 preferred.
• Apache Hadoop
• Cloudera
• HDFS
• YARN
• Hive
• HBase

Development

• Scala 2.12
• Functional Programming
• ZIO
• Cats

Streaming & Integration

• Distributed Data Processing

Security

• Work within secured Big Data environments.
• Apply and troubleshoot security mechanisms including:
• Ensure solutions comply with platform access and security requirements.
• Collaborate with infrastructure teams when investigating authorization and authentication issues.

Scheduling

• Schedule and monitor data processing workloads using Control-M.
• Support dependencies between scheduled jobs and data-processing workflows.
• Investigate failed executions and operational issues.
• Contribute to stable application operation across production environments.

DevOps

• Jenkins
• XL Deploy
• XL Release
• CI/CD

Development Tools

• IntelliJ IDEA
• Git
• Bitbucket
• Jira
• Confluence
• Scrum

Nice to Have

• Experience with ZIO or Cats.
• Strong Functional Programming knowledge in Scala.
• Experience modernizing large Big Data applications.
• Knowledge of Spark performance analysis in high-volume environments.
• Experience supporting production-critical data pipelines.
• Knowledge of automated testing for Scala/Spark applications.
• Experience with Cloudera platform administration or performance troubleshooting.
• Familiarity with AI-assisted development tools.

The role requires regular interaction with stakeholders with different levels of technical expertise, so the ability to explain complex topics clearly is important.

• Strong analytical and problem-solving capabilities.
• Ability to understand complex technical environments.
• Comfortable estimating and planning development activities.
• Strong communication with technical and non-technical stakeholders.
• Proactive and collaborative mindset.
• Strong attention to code quality and maintainability.
• Ability to take ownership of technically complex topics.
• Work with Spark and Scala on large-scale enterprise data solutions.
• Gain deep exposure to a complete Cloudera/Hadoop ecosystem.
• Solve complex distributed-processing and performance challenges.
• Work with high-volume data pipelines and Kafka integrations.
• Influence application architecture through component reuse and modernization.
• Combine Big Data engineering with DevOps, security, and production delivery.
• Join a team where code quality, technical collaboration, and continuous improvement are strongly valued.