Senior Data Consultant (Hadoop, Spark) (Kraków, PL, 30-302)

GFT Technologies SE; Polska; Hybrid; PLN 172,000 - 225,000; Full time

Job summary

GFT Technologies SE is seeking a data engineer to join a technical team alongside Engineers, Data Analysts and Business Analysts, contributing to an Agile development process while designing, developing and maintaining scalable data solutions in a dynamic DevOps environment.

You will work on tasks including PySpark/Scala development, automating tests, code reviews, production support and building scalable data pipelines.

Qualifications

• Experience with Pyspark or Scala development and design.
• Experience using scheduling tools such as Airflow.
• Knowledge of Hadoop ecosystem including Spark, Hive, YARN and ETL frameworks.
• Strong SQL and RESTful services knowledge.
• Experience working on Unix or Linux platforms.
• Hands-on experience building data pipelines using Hadoop components.
• Experience with Git, GitHub, Jenkins, Ansible and JIRA.
• Understanding of big data modelling using relational and non-relational techniques.
• Experience debugging code and communicating findings to development teams.
• Openness to work 2 days a week from our client's office (Kraków).

Responsibilities

• Define and contribute to software design and development using Pyspark.
• Automate testing of new and existing components.
• Promote development standards through code reviews and mentoring.
• Provide production support and troubleshooting.
• Implement tools and processes ensuring performance, scalability and monitoring.
• Collaborate with Business Analysts to interpret and implement requirements.
• Participate in planning, sprint reviews and retrospectives.
• Contribute to system architecture and design.

Skills

Pyspark/Scala development

Airflow

Hadoop ecosystem

SQL

Unix/Linux

Data pipelines

Git/GitHub/Jenkins/Ansible/JIRA

Big data modeling

Debugging code

Kraków on-site (2 days)

Tools

Airflow

Hadoop

Spark

Hive

YARN

ETL frameworks

Git

GitHub

Jenkins

Ansible

JIRA

Job description

Type of contract: B2B contract
Salary range: 125-163 PLN net/h

What will you do?

You will work as a key member of a technical team alongside Engineers, Data Analysts and Business Analysts, contributing to a collaborative Agile development process while designing, developing and maintaining scalable data solutions in a dynamic DevOps environment.

Your tasks

• Define and contribute to software design and development using Pyspark


• Automate testing of new and existing components


• Promote development standards through code reviews and mentoring


• Provide production support and troubleshooting


• Implement tools and processes ensuring performance, scalability and monitoring


• Collaborate with Business Analysts to interpret and implement requirements


• Participate in planning, sprint reviews and retrospectives


• Contribute to system architecture and design

Requirements

• Experience with Pyspark or Scala development and design


• Experience using scheduling tools such as Airflow


• Knowledge of Hadoop ecosystem including Spark, Hive, YARN and ETL frameworks


• Strong SQL and RESTful services knowledge


• Experience working on Unix or Linux platforms


• Hands-on experience building data pipelines using Hadoop components


• Experience with Git, GitHub, Jenkins, Ansible and JIRA


• Understanding of big data modelling using relational and non-relational techniques


• Experience debugging code and communicating findings to development teams


• Openness to work 2 days a week from our client's office (Kraków)

Nice to have

• Experience with Elasticsearch


• Experience developing Java APIs


• Experience in data ingestion processes


• Understanding of cloud design patterns


• Exposure to DevOps and Agile methodologies such as Scrum and Kanban


• Experience with Spark streaming


• Experience with Apache Airflow in production


• Experience with Hadoop ecosystem in enterprise environments


• Knowledge of Python backend services


• Experience with Scala for high performance systems


• Experience in data integration and ETL processes


• Knowledge of PL/SQL


• Experience with Linux and Unix system operations