PySpark Developer

WIRE IT; Lisboa; Híbrido; EUR 55 000 - 90 000; Tempo integral

Vantagens oferecidas por esta oferta de emprego

Health Insurance

22 days vacation

4 extra days annually

Meal Allowance on card (Coverflex)

Referral Bonus

Special discounts and flex options

Training annual budget

Great team-oriented culture

Chance to join growing international公司

Challenging projects

Challenging projects giving the unique

Resumo da oferta

Wire IT in Portugal is seeking a Senior PySpark Developer to join our experienced data engineering team. Based in Porto or Lisbon, this hybrid role focuses on building scalable batch and streaming data pipelines with PySpark, Python and SQL, optimizing Spark architecture and performance.

You will work on challenging projects for an international client base in an Agile environment, with tools like Airflow, Databricks and Git.

Qualificações

• At least 5 years of experience in software development or data engineering.
• Strong professional experience with PySpark and Apache Spark.
• Advanced proficiency in Python and SQL.
• Experience developing scalable batch and streaming data pipelines.
• Deep understanding of Spark architecture, DAGs, Lazy Evaluation, caching, partitioning, shuffling, memory tuning, and data skew.
• Experience integrating SQL/NoSQL databases, APIs, files, and Kafka streams.
• Knowledge of Parquet, Delta Lake, and Avro.
• Experience with AWS, Azure, or GCP.
• Experience with Databricks, EMR, or similar platforms.
• Experience with Apache Airflow, Prefect, or similar orchestration tools.
• Proficiency with Git and CI/CD pipelines.
• Experience with PyTest, unit testing, integration testing, and Spark data mocking.
• Good communication skills and experience working in Agile environments.
• Strong analytical, problem-solving, and performance-optimization capabilities.

Conhecimentos

5+ years experience

PySpark

Apache Spark

Python

SQL

Batch and streaming pipelines

Spark architecture understanding

SQL/NoSQL integration

Kafka streams

Parquet/Delta Lake/Avro

AWS/Azure/GCP

Databricks/EMR

Airflow/Prefect

Git/CI-CD

Testing (PyTest)

Agile methodologies

Communication skills

Ferramentas

Databricks

Airflow

Prefect

Git

Descrição da oferta de emprego

Based in Portugal, Wire IT is your specialized IT consulting partner with 18 years of experience driven by an experienced and senior team that helps clients make the right decisions in a fast-moving market.

Wire IT's ambition is to grow while keeping true to its nature: agile, people-centered, and fun. As we like to say: Its not only what we do, its how we manage that truly make the difference.

If you're looking for more than just a job - if you're looking for a place where you can truly make a difference - you might be the next person to join WIRE IT.

Position: Senior PySpark Developer

Location: Porto or Lisbon, Portugal (Hybrid)

Required Skills:

• At least 5 years of experience in software development or data engineering;


• Strong professional experience with PySpark and Apache Spark;


• Advanced proficiency in Python and SQL;


• Experience developing scalable batch and streaming data pipelines;


• Deep understanding of Spark architecture, DAGs, Lazy Evaluation, caching, partitioning, shuffling, memory tuning, and data skew;


• Experience integrating SQL/NoSQL databases, APIs, files, and Kafka streams;


• Knowledge of Parquet, Delta Lake, and Avro;


• Experience with AWS, Azure, or GCP;


• Experience with Databricks, EMR, or similar platforms;


• Experience with Apache Airflow, Prefect, or similar orchestration tools;


• Proficiency with Git and CI/CD pipelines;


• Experience with PyTest, unit testing, integration testing, and Spark data mocking;


• Good communication skills and experience working in Agile environments;


• Strong analytical, problem-solving, and performance-optimization capabilities.

We offer:

• No-term full-time contract;


• Health Insurance;


• 22 days of paid vacation;


• 4 extra days annually (Carnival, Christmas Eve, New Year's Eve, Birthday);


• Meal Allowance on card (Coverflex);


• Referral Bonus;


• Special discounts and flex options;


• Training annual budget;


• A great team-oriented culture;


• Chance to become a part of a dynamically growing international company;


• Challenging projects giving the unique opportunity to grow and learn.

Following the EU General Data Protection Regulation No. 2016/679, we inform you that by responding to this announcement, you consent to the use of your data by Wire IT, for as long as necessary. In case of doubt or request for elimination of your personal data, please send an email to [e-mail].