Big Data Engineer (Spark/Scala) Spark, Scala, Python, Hadoop, SQL, AWS Warsaw, Łódź, Gdańsk, Gdynia

Diverse CG Sp. z o.o. Sp.k.; Łódź, Gdynia; On-site; PLN 80,000 - 100,000; Full time

Benefits offered by this job

Private medical care

Co-financing for the sports card

Constant support of dedicated consultant

Job summary

Diverse CG Sp. z o.o. Sp.k. in Łódź, Poland, is seeking a Big Data Engineer with extensive experience in Spark and Scala. The role involves building distributed data processing pipelines, and collaborating with cross-functional teams in a Continuous Delivery environment. Ideal candidates will have strong skills in Python, CI/CD tools, and SQL. Benefits include private medical care and co-financing for a sports card. Candidates should demonstrate a collaborative mindset and be self-motivated.

Qualifications

• 7+ years of experience in Spark & Scala.
• 5+ years of development experience with Python.
• 3+ years experience with CI/CD tools such as Jenkins.

Responsibilities

• Build distributed and highly parallelized BigData processing pipeline.
• Leverage Spark for data enrichment and transformation.
• Work closely with DevOps, QA, and Product Management teams.

Skills

Spark & Scala

Python

Robot Framework

CI/CD tools

AWS

SQL

BitBucket and GIT

Linux and bash

Tools

Jenkins

Bamboo

Jira

Job description

As a recruitment company, DCG understands that every business is powered by experienced professionals. Our management style and partnership approach enable us to meet your needs and provide continuous support. Due to our ongoing growth and the large number of recruitment projects we undertake for our partners, we are currently looking for:

Responsibilities:

• Building distributed and highly parallelized BigData processing pipeline which process massive amount of data (both structured and unstructured) in near real-time
• Leverage Spark to enrich and transform corporate data to enable searching, data visualization, and advanced analytics
• Requirement analysis and questioning requirements when they are not clear
• Work closely with DevOps, QA and Product Management teams in a Continuous Delivery environment

Requirements:

• 7+ years of experience in Spark & Scala
• 5+ years of development experience with Python
• 3+ years experience with Robot Framework
• 3+ years experience with CI/CD tools such as Jenkins
• Experience with distributed data processing engines like Spark
• Experience with Hadoop and (Hive, Oozie, map reduce, ect)
• Strong SQL skills with previous experience in creating data flows
• BitBucket and GIT expert
• AWS knowledge about architectural components ( Lamba, step functions, state machine, ect)
• Unit testing Experience (Junit 5, Mockito, Spark testing)
• Code versioning strategy & Branching strategy
• Advanced English skills
• Strong working experience with Linux and bash
• Experience using following tools: Jenkins or Bamboo, Jira, bitbucket
• Demonstrate a collaborative mindset, achieving goals through effective communication and teamwork within an Agile (SAFe) environment
• Show curiosity and responsiveness, with the ability to understand others’ needs to ensure delivery of the expected outcomes
• Maintain a strong focus on quality, continuously striving to improve ways of working and delivered solutions
• Exhibit a high level of self‑motivation and ownership in day‑to‑day work

Nice to have:

• Experience with containerization using Docker
• Knowledge of streaming and queues technologies e.g. KAFKA, IBM MQ etc
• Working knowledge of DevOps

Offer:

• Private medical care
• Co‑financing for the sports card
• Constant support of dedicated consultant

Personal Consulting Agency (License No. 4642)