Big Data Engineer (Spark/Scala) Spark, Scala, Python, Hadoop, SQL, AWS Warsaw, Łódź, Gdańsk, Gdynia

DCG Poland; Łódź, Gdynia; On-site; PLN 167,400 - 279,000; Full time

Benefits offered by this job

Private medical care

Co-financing for the sports card

Constant support of dedicated consultant

Job summary

DCG Poland is looking for experienced professionals to develop distributed BigData processing pipelines. Candidates should have 7+ years of expertise in Spark and Scala, along with substantial experience in Python and Robot Framework. You will collaborate with DevOps and QA teams in an Agile environment, maintaining a focus on quality and continuous improvement. The role offers private medical care and support from a dedicated consultant.

Qualifications

• 7+ years of experience in Spark & Scala.
• 5+ years of development experience with Python.
• 3+ years experience with Robot Framework.
• 3+ years experience with CI/CD tools such as Jenkins.
• Strong SQL skills with previous experience in creating data flows.

Responsibilities

• Building distributed and highly parallelized BigData processing pipeline.
• Leverage Spark for data transformation.
• Work closely with DevOps, QA and Product Management teams.

Skills

Spark & Scala

Python

Robot Framework

CI/CD tools (Jenkins)

Hadoop

SQL

BitBucket and GIT

AWS

Linux and bash

Jira

Docker (nice to have)

KAFKA, IBM MQ (nice to have)

Tools

Jenkins or Bamboo

Jira

bitbucket

Job description

As a recruitment company, DCG understands that every business is powered by experienced professionals. Our management style and partnership approach enable us to meet your needs and provide continuous support. Due to our ongoing growth and the large number of recruitment projects we undertake for our partners, we are currently looking for:

Responsibilities:

• Building distributed and highly parallelized BigData processing pipeline which process massive amount of data (both structured and unstructured) in near real-time
• Leverage Spark to enrich and transform corporate data to enable searching, data visualization, and advanced analytics
• Requirement analysis and questioning requirements when they are not clear
• Work closely with DevOps, QA and Product Management teams in a Continuous Delivery environment

Requirements:

• 7+ years of experience in Spark & Scala
• 5+ years of development experience with Python
• 3+ years experience with Robot Framework
• 3+ years experience with CI/CD tools such as Jenkins
• Experience with distributed data processing engines like Spark
• Experience with Hadoop and (Hive, Oozie, map reduce, ect)
• Strong SQL skills with previous experience in creating data flows
• BitBucket and GIT expert
• AWS knowledge about architectural components ( Lamba, step functions, state machine, ect)
• Unit testing Experience (Junit 5, Mockito, Spark testing)
• Code versioning strategy & Branching strategy
• Advanced English skills
• Strong working experience with Linux and bash
• Experience using following tools: Jenkins or Bamboo, Jira, bitbucket
• Demonstrate a collaborative mindset, achieving goals through effective communication and teamwork within an Agile (SAFe) environment
• Show curiosity and responsiveness, with the ability to understand others’ needs to ensure delivery of the expected outcomes
• Maintain a strong focus on quality, continuously striving to improve ways of working and delivered solutions
• Exhibit a high level of self‑motivation and ownership in day‑to‑day work

Nice to have:

• Experience with containerization using Docker
• Knowledge of streaming and queues technologies e.g. KAFKA, IBM MQ etc
• Working knowledge of DevOps

Offer:

• Private medical care
• Co‑financing for the sports card
• Constant support of dedicated consultant

Personal Consulting Agency (License No. 4642)