Data Platform Engineer, Mid

Jobtailor; Lisboa; Presencial; EUR 45 000 - 65 000; Tempo integral

Resumo da oferta

Jobtailor in Lisbon seeks an experienced Data Engineer to design, develop and optimize data pipelines using Spark and Spark Streaming.

You will write scalable Python following PEP 8, work with SQL for data modeling and analytics, and use Databricks for notebooks, clusters and Delta Lake.

Collaborating with diverse teams, you will apply DevOps practices with GitHub Actions and CI/CD, with a strong focus on delivery and continuous improvement.

Qualificações

• Strong Python (OOP) skills and adherence to PEP 8.
• Extensive experience with Spark and Spark Streaming.
• Proficient in SQL for data modeling and analytics.
• Hands-on Databricks experience: notebooks, clusters, Delta Lake.
• Familiarity with DevOps; GitHub Actions and CI/CD tooling.

Responsabilidades

• Develop, optimize and monitor data pipelines using Spark and Spark Streaming.
• Write scalable Python code following PEP 8 with unit and integration tests.
• Model data with SQL for analytics and integrate with analytical systems.
• Use Databricks for notebooks, clusters and Delta Lake.
• Collaborate with cross-functional teams and communicate effectively.

Conhecimentos

Python (OOP)

Spark

SQL

Databricks

Delta Lake

GitHub Actions

CI/CD

Azure / GCP

Unit & Integration Testing

Agile / Delivery

Descrição da oferta de emprego

Responsibilities

• Develop, implement and optimize data pipelines using Spark and Spark Streaming
• Write scalable, object-oriented Python code following best practices (PEP 8), with unit and integration tests
• Work with SQL for data modeling, query optimization and integration with analytical systems
• Use Databricks for managing clusters, notebooks and Delta Lake
• Collaborate with technical and non-technical teams, facilitating communication and understanding of requirements
• Implement DevOps practices and deployment automation using GitHub and GitHub Actions
• Ensure code quality and work in Agile environments

Requirements

• At least 3 years of experience in similar roles
• Advanced proficiency in Python with focus on OOP, best practices (PEP 8), unit and integration testing, and code optimization
• Experience in SQL for modeling, query optimization and integration with analytical systems
• Advanced knowledge of Spark and Spark Streaming for batch and streaming processing and performance optimization
• Experience with Databricks (notebooks, clusters, Delta Lake)
• Solid knowledge of Azure ecosystems (preferred) or GCP
• Experience with DevOps methodologies and CI/CD tools, especially GitHub and GitHub Actions
• Ability to understand requirements and technical solutions
• Strong communication skills across diverse audiences
• Critical thinking and focus on continuous improvement
• Strong delivery orientation with focus on meeting deadlines and objectives
• Ability to solve complex problems and strong analytical thinking