Senior PySpark Developer: Scalable Data Pipelines

Wire-IT; Lisboa; Híbrido; EUR 60 000 - 80 000; Tempo integral

Vantagens oferecidas por esta oferta de emprego

Health Insurance

22 days vacation

4 extra days annually (Carnival, X-mas

Meal Allowance on card (Coverflex)

Referral Bonus

Discounts and flexible options

Annual training budget

Team-oriented culture

Growth opportunities

Challenging projects

Resumo da oferta

Wire IT, based in Portugal, is seeking a Senior PySpark Developer to join a senior, agile team. You will design and optimize data pipelines using PySpark and Spark, connecting SQL/NoSQL, Kafka streams, Parquet/Delta Lake, and cloud platforms.

Hybrid work in Porto or Lisbon, Portugal. 5+ years of software/data engineering experience required. Strong Python, SQL and performance tuning skills are essential for this role.

Qualificações

• Minimum 5 years in software development or data engineering.
• Strong experience with PySpark and Apache Spark.
• Advanced Python and SQL skills.
• Experience building scalable batch and streaming data pipelines.
• Deep understanding of Spark internals: DAGs, lazy evaluation, caching, partitioning, shuffling, memory tuning, data skew.
• Experience integrating SQL/NoSQL, APIs, files and Kafka streams.
• Knowledge of Parquet, Delta Lake and Avro.
• Experience with AWS, Azure or GCP.
• Experience with Databricks, EMR or similar platforms.
• Experience with Airflow, Prefect or similar orchestration tools.
• Proficiency with Git and CI/CD.
• Experience with PyTest and testing in Spark environments.
• Good communication and Agile experience.
• Strong analytical and problem-solving abilities.

Conhecimentos

PySpark

Python

SQL

Spark Architecture

Data Pipelines

Kafka Streams

Cloud Platforms

CI/CD

PyTest

Agile

Communication

Performance Tuning

Ferramentas

Databricks

Airflow

Delta Lake

EMR

Descrição da oferta de emprego

Wire IT, based in Portugal, is seeking a Senior PySpark Developer to join a senior, agile team. You will design and optimize data pipelines using PySpark and Spark, connecting SQL/NoSQL, Kafka streams, Parquet/Delta Lake, and cloud platforms.

Hybrid work in Porto or Lisbon, Portugal. 5+ years of software/data engineering experience required. Strong Python, SQL and performance tuning skills are essential for this role.