PySpark Data Engineer — Big Data & Streaming

Hexa Consulting; Porto; Presencial; EUR 22 000 - 28 000; Tempo integral

Vantagens oferecidas por esta oferta de emprego

Private health insurance

Mental health support

Training opportunities

Flexible work

Childcare vouchers

Team events

Upskilling in AI governance

Resumo da oferta

Hexa Consulting is seeking a PySpark Developer/Data Engineer to design, develop, and optimize large-scale data processing solutions for a financial services client in Porto. You will build scalable batch and streaming pipelines using PySpark and Spark, optimize workloads, and ensure data quality across diverse sources.

The role emphasizes translating requirements into efficient designs, adopting best practices, and collaborating with cross-functional teams in an Agile setup.

Qualificações

• Proven PySpark/Data Engineer background with Spark expertise.
• Strong problem solving, autonomy, and data quality focus.

Responsabilidades

• Design, develop and maintain scalable batch and streaming data pipelines.
• Optimize Spark workloads: partitioning, shuffle reduction, memory tuning and data skew mitigation.
• Integrate and validate large heterogeneous datasets from databases, APIs, files and streams.
• Translate requirements into data flows and technical designs.
• Apply software engineering best practices: clean code, reviews, tests, maintainable delivery.

Conhecimentos

Python

PySpark

Spark SQL

DataFrames & Datasets

Spark architecture

SQL (complex queries)

Parquet

SQL/NoSQL databases

APIs

Kafka streams

Git

PyTest

Unit/Integration testing

Databricks or EMR

Ferramentas

Databricks

EMR

Descrição da oferta de emprego

Hexa Consulting is seeking a PySpark Developer/Data Engineer to design, develop, and optimize large-scale data processing solutions for a financial services client in Porto. You will build scalable batch and streaming pipelines using PySpark and Spark, optimize workloads, and ensure data quality across diverse sources.

The role emphasizes translating requirements into efficient designs, adopting best practices, and collaborating with cross-functional teams in an Agile setup.