Senior Data Consultant (Hadoop, Spark)

GFT TECHNOLOGIES SE; Kraków; On-site; PLN 172,200 - 224,548; Part time

Job summary

GFT TECHNOLOGIES SE is seeking a Senior Data Consultant with expertise in Pyspark and the Hadoop ecosystem. The role involves designing and maintaining scalable data solutions while working within a hybrid environment in Kraków.

Your responsibilities include contributing to Agile processes, software development, and collaborating with various teams. Ideal candidates will have experience with data pipelines, ETL frameworks, and cloud design patterns.

Qualifications

• Experience with Pyspark or Scala development and design.
• Experience using scheduling tools such as Airflow.
• Knowledge of Hadoop ecosystem including Spark, Hive, YARN and ETL frameworks.

Responsibilities

• Define and contribute to software design and development using Pyspark.
• Automate testing of new and existing components.
• Collaborate with Business Analysts to interpret and implement requirements.

Skills

Pyspark or Scala development

Hadoop ecosystem

SQL

RESTful services

Git, GitHub

Jenkins, Ansible

Unix or Linux

ETL frameworks

Tools

Airflow

Elasticsearch

Apache

Job description

Senior Data Consultant (Hadoop, Spark)

Location: Kraków, PL, 30-302

Working place: Hybrid

Type of contract: B2B contract
Salary range: 125-163 PLN net/h

What will you do?

You will work as a key member of a technical team alongside Engineers, Data Analysts and Business Analysts, contributing to a collaborative Agile development process while designing, developing and maintaining scalable data solutions in a dynamic DevOps environment.

• Define and contribute to software design and development using Pyspark
• Automate testing of new and existing components
• Promote development standards through code reviews and mentoring
• Provide production support and troubleshooting
• Implement tools and processes ensuring performance, scalability and monitoring
• Collaborate with Business Analysts to interpret and implement requirements
• Participate in planning, sprint reviews and retrospectives
• Contribute to system architecture and design

Requirements

• Experience with Pyspark or Scala development and design
• Experience using scheduling tools such as Airflow
• Knowledge of Hadoop ecosystem including Spark, Hive, YARN and ETL frameworks
• Strong SQL and RESTful services knowledge
• Experience working on Unix or Linux platforms
• Hands-on experience building data pipelines using Hadoop components
• Experience with Git, GitHub, Jenkins, Ansible and JIRA
• Understanding of big data modelling using relational and non-relational techniques
• Experience debugging code and communicating findings to development teams
• Openness to work 2 days a week from our client's office (Kraków)

Nice to have

• Experience with Elasticsearch
• Understanding of cloud design patterns
• Exposure to DevOps and Agile methodologies such as Scrum and Kanban
• Experience with Spark streaming
• Experience with Apache Airflow in production
• Experience with Hadoop ecosystem in enterprise environments
• Knowledge of Python backend services
• Experience with Scala for high performance systems
• Experience in data integration and ETL processes
• Knowledge of PL/SQL
• Experience with Linux and Unix system operations