BigData Engineer (Python or Scala + Spark)

Integral Solutions - Informatica Distributor; Warszawa; On-site; PLN 223,200 - 334,800; Full time

Job summary

A leading IT service provider is seeking a skilled BigData Engineer for an exciting project with a major banking client. This opportunity is based in Warsaw and involves building and enhancing Big Data solutions. The ideal candidate has a minimum of 3 years of experience in Python or Scala, expertise in Spark, and familiarity with data warehouse principles. The position requires good engineering practices and the ability to work with various data formats.

Qualifications

• Minimum 3 years of experience in Python or Scala programming.
• Commercial experience with BigData technologies; Spark is a must.
• Understanding of database usage scenarios.

Responsibilities

• Write and maintain data pipelines and applications using Python or Scala.
• Work with Apache Spark to process large amounts of data.
• Combine data from different sources into a single system.

Skills

Python programming

Scala programming

Apache Spark

SQL

Data Warehouse principles

Agile processes

Kafka

Tools

Maven

Jira

Confluence

Job description

BigData Engineer (Python or Scala + Spark)

We are looking for a skilled BigData Engineer to join an exciting project for a leading banking client, one of the top players in the Polish market. You will work on building and improving Big Data solutions that support key business processes, using modern technologies and best engineering practices.

It's a Warsaw-based opportunity. The team visit the office once in two weeks.

Responsibilities:

• Write and maintain data pipelines and applications using Python or Scala
• Work with Apache Spark and other Big Data tools to process large amounts of data
• Use data warehouse principles to keep data well-structured and reliable
• Follow good engineering practices: clear code, proper testing, documentation, and clean design
• Handle different data formats like JSON, Parquet, ORC, and Avro
• Write and improve SQL queries to work with data efficiently
• Combine data from different sources into a single system

You will be perfect candidate if you have:

• Minimum 3 years of experience in Python or Scala programming
• Commercial experience and exposure to BigData technologies, Spark is a must
• Familiarity with Data Warehouse principles
• Knowledge of good engineering practices, Big Data processing, including Design standards, Data Modelling techniques, coding, documentation, testing and implementation
• Experience with various data formats: JSON, PARQUET, ORC, AVRO
• Understanding database types and usage scenarios, e.g. Hive, Kudu, HBase, Iceberg, etc.
• Advanced knowledge of SQL Experience in integrating data from multiple data sources
• Knowledge of tools for building projects/applications, e.g. Maven
• Advanced knowledge of Polish and good knowledge of English

It will be a strong plus to have:

• Practical knowledge of Agile processes and tools: Jira, Confluence, Kanban, Scrum, etc.
• Knowledge of the Kubeflow platform
• Knowledge of streaming technologies and tools such as Kafka, Apache Nifi