Senior Spark & Big Data Platform Engineer

act digital; Portugal; On-site; EUR 60,000 - 90,000; Full time

Job summary

act digital is seeking a Big Data Engineer in Portugal to develop features on the Big Data platform and optimize Spark applications for large-scale data processing. You will contribute to a scalable architecture, participate in code reviews, and drive CI/CD deployments in a DevOps context.

Ideal candidates have strong Spark 3 and Scala 2.12 skills, plus experience with Hadoop ecosystem tools and on-premises environments. Collaboration with cross-functional teams is essential.

Qualifications

• 3+ years of experience in a similar field (as stated).
• Strong experience with Spark 3 and large-scale data processing.
• Proficiency in Scala 2.12 for Spark development.
• Solid knowledge of coding best practices and clean code standards.
• Experience with Hadoop ecosystem components (YARN, HDFS, Hive, HBase, Kafka).
• Knowledge of Kerberos and Apache Ranger for Big Data security.
• DevOps experience with CI/CD pipelines and deployment tools.
• Familiarity with Jira and Confluence for project management.
• Experience with on-premises Big Data environments (preferably Cloudera).
• Experience with version control (Git/Bitbucket) and job scheduling (Control-M).
• Nice to have knowledge of ZIO or Cats.

Responsibilities

• Develop new features within the Big Data platform.
• Optimize Spark applications for performance and scalability.
• Rationalize and reuse components for a scalable architecture.
• Contribute to code reviews and maintain code quality.
• Participate in deployments to testing and production in a DevOps context.
• Manage upgrades and related testing/validation processes.
• Write and maintain technical documentation as required.
• Plan and estimate development tasks with dependencies.
• Participate in Agile/Scrum ceremonies and use Jira/Confluence for collaboration.
• Collaborate with infrastructure teams, developers, and business analysts.

Skills

Apache Spark 3

Scala 2.12

Big Data security (Kerberos, Ranger)

Hadoop ecosystem (YARN, HDFS, Hive, H-

Tools

Jenkins

XL Deploy

XL Release

Git

Bitbucket

Control-M

Jira

Confluence

Cloudera

Job description

act digital is seeking a Big Data Engineer in Portugal to develop features on the Big Data platform and optimize Spark applications for large-scale data processing. You will contribute to a scalable architecture, participate in code reviews, and drive CI/CD deployments in a DevOps context.

Ideal candidates have strong Spark 3 and Scala 2.12 skills, plus experience with Hadoop ecosystem tools and on-premises environments. Collaboration with cross-functional teams is essential.