Senior Data Engineer (Hadoop, Spark)

GFT Technologies Poland; Kraków; Hybrid; PLN 176,328 - 271,188; Full time

Benefits offered by this job

Hybrid work in Krakow

Medical, sport, lunch subsidy, life保险

Job summary

GFT Technologies Poland in Kraków is seeking a data engineer to join a technical team of engineers, data analysts and business analysts. You will design, develop and maintain scalable data solutions in a dynamic DevOps environment.

You will work with PySpark/Scala, Hadoop ecosystem, Airflow and SQL, building robust data pipelines, performing code reviews, mentoring, and providing production support. The role offers hybrid work in Kraków and a benefits package.

Qualifications

• Experience with PySpark or Scala development and design.
• Experience using scheduling tools such as Airflow.
• Knowledge of Hadoop ecosystem including Spark, Hive, YARN and ETL frameworks.
• Strong SQL and RESTful services knowledge.
• Experience working on Unix or Linux platforms.
• Hands-on experience building data pipelines using Hadoop components.
• Experience with Git, GitHub, Jenkins, Ansible and JIRA.
• Understanding of big data modelling using relational and non-relational techniques.
• Experience debugging code and communicating findings to development teams.

Responsibilities

• Define and contribute to software design and development using PySpark.
• Automate testing of new and existing components.
• Mentor team members and promote coding standards through reviews.
• Provide production support and troubleshooting.
• Implement tools and processes for performance, scalability and monitoring.
• Collaborate with Business Analysts to interpret requirements.
• Participate in planning, sprint reviews and retrospectives.
• Contribute to system architecture and design.

Skills

PySpark

Scala

Hadoop ecosystem

Hive

YARN

ETL

SQL

RESTful services

Unix/Linux

Git

Jenkins

Airflow

Ansible

JIRA

Python

PL/SQL

Tools

Airflow

Git

GitHub

Jenkins

Ansible

JIRA

Job description

Salary range: 15800-24300 PLN gross/month

What will you do?

You will work as a key member of a technical team alongside Engineers, Data Analysts and Business Analysts, contributing to a collaborative Agile development process while designing, developing and maintaining scalable data solutions in a dynamic DevOps environment.

• Define and contribute to software design and development using Pyspark
• Automate testing of new and existing components
• Promote development standards through code reviews and mentoring
• Provide production support and troubleshooting
• Implement tools and processes ensuring performance, scalability and monitoring
• Collaborate with Business Analysts to interpret and implement requirements
• Participate in planning, sprint reviews and retrospectives
• Contribute to system architecture and design
• Experience with Pyspark or Scala development and design
• Experience using scheduling tools such as Airflow
• Knowledge of Hadoop ecosystem including Spark, Hive, YARN and ETL frameworks
• Strong SQL and RESTful services knowledge
• Experience working on Unix or Linux platforms
• Hands‑on experience building data pipelines using Hadoop components
• Experience with Git, GitHub, Jenkins, Ansible and JIRA
• Understanding of big data modelling using relational and non-relational techniques
• Experience debugging code and communicating findings to development teams

Nice to have

• Experience with Elasticsearch
• Understanding of cloud design patterns
• Exposure to DevOps and Agile methodologies such as Scrum and Kanban
• Experience with Spark streaming
• Experience with Apache Airflow in production
• Experience with Hadoop ecosystem in enterprise environments
• Knowledge of Python backend services ??? Wait, we ended prematurely. Let me finish correctly.
• Knowledge of Python backend services
• Experience with Scala for high performance systems
• Experience in data integration and ETL processes
• Knowledge of PL/SQL
• Experience with Linux and Unix system operations

We offer

• Hybrid work in Krakow (2 office days per week)
• Working in a highly experienced and dedicated team
• Benefit package tailored to your needs (medical, sport, lunch subsidy, life insurance, etc.)