Big Data Developer

act digital; Portugal; Presencial; EUR 60.000 - 90.000; Jornada completa

Descripción de la vacante

act digital is seeking a Big Data Engineer in Portugal to develop features on the Big Data platform and optimize Spark applications for large-scale data processing. You will contribute to a scalable architecture, participate in code reviews, and drive CI/CD deployments in a DevOps context.

Ideal candidates have strong Spark 3 and Scala 2.12 skills, plus experience with Hadoop ecosystem tools and on-premises environments. Collaboration with cross-functional teams is essential.

Formación

• 3+ years of experience in a similar field (as stated).
• Strong experience with Spark 3 and large-scale data processing.
• Proficiency in Scala 2.12 for Spark development.
• Solid knowledge of coding best practices and clean code standards.
• Experience with Hadoop ecosystem components (YARN, HDFS, Hive, HBase, Kafka).
• Knowledge of Kerberos and Apache Ranger for Big Data security.
• DevOps experience with CI/CD pipelines and deployment tools.
• Familiarity with Jira and Confluence for project management.
• Experience with on-premises Big Data environments (preferably Cloudera).
• Experience with version control (Git/Bitbucket) and job scheduling (Control-M).
• Nice to have knowledge of ZIO or Cats.

Responsabilidades

• Develop new features within the Big Data platform.
• Optimize Spark applications for performance and scalability.
• Rationalize and reuse components for a scalable architecture.
• Contribute to code reviews and maintain code quality.
• Participate in deployments to testing and production in a DevOps context.
• Manage upgrades and related testing/validation processes.
• Write and maintain technical documentation as required.
• Plan and estimate development tasks with dependencies.
• Participate in Agile/Scrum ceremonies and use Jira/Confluence for collaboration.
• Collaborate with infrastructure teams, developers, and business analysts.

Conocimientos

Apache Spark 3

Scala 2.12

Big Data security (Kerberos, Ranger)

Hadoop ecosystem (YARN, HDFS, Hive, H-

Herramientas

Jenkins

XL Deploy

XL Release

Git

Bitbucket

Control-M

Jira

Confluence

Cloudera

Descripción del empleo

• Between 3 and 6 years of experience in a similar field;
• Strong experience in Apache Spark 3, including application optimization techniques and large-scale data processing.
• Proficiency in Scala 2.12, particularly for Spark application development.
• Solid knowledge of coding best practices, clean code principles and software development standards.
• Very good knowledge of the Hadoop ecosystem, including YARN, HDFS, Hive, HBase and Kafka.
• Knowledge of Big Data security mechanisms, particularly Kerberos and Apache Ranger.
• Experience implementing and maintaining CI/CD pipelines in a DevOps environment.
• Knowledge of deployment and automation tools, namely Jenkins, XL Deploy and XL Release.
• Experience with version control systems, particularly Bitbucket/Git.
• Knowledge of job scheduling and orchestration tools, particularly Control-M.
• Experience working with on-premises Big Data environments, preferably Cloudera.
• Knowledge of functional programming frameworks such as ZIO or Cats is considered an advantage.

Responsibilities

• Develop new features and components within the Big Data platform.
• Optimize Spark applications to ensure performance, efficiency and scalability when processing large volumes of data.
• Rationalize, optimize and reuse existing components to build a consistent, scalable and non-redundant architecture.
• Participate in code reviews, ensuring code quality, readability, maintainability and consistency.
• Participate in deployments to testing and production environments within a DevOps context.
• Manage component upgrades, including the associated testing and validation processes.
• Implement and maintain CI/CD pipelines.
• Write and maintain technical documentation whenever required.
• Plan and estimate development tasks, identifying technical requirements and dependencies.
• Participate in Agile/Scrum ceremonies, including daily meetings, sprint planning, sprint reviews and retrospectives.
• Use collaboration and project management tools such as Jira and Confluence.
• Collaborate with Infrastructure teams, Developers, Business Analysts and end users, adapting communication to different technical audiences.
• Contribute to continuous improvement initiatives, enhancing development processes, architecture and overall code quality.