Data Engineer - Senior / Manager

Us3 Consulting; Warszawa; On-site; PLN 180,000 - 300,000; Full time

Job summary

Us3 Consulting is seeking an experienced Data Engineer at Manager level to design, develop and optimise modern data ingestion and processing solutions.

You will work across teams to build reliable, scalable and traceable data pipelines, while continuously improving data products and services, using Azure Data Factory and Azure Databricks.

Qualifications

• 8+ years experience in data engineering at Manager/Senior level.
• Azure Data Factory: strong hands-on experience developing pipelines, activities, triggers and data flows.
• Azure Databricks & Apache Spark: strong experience with notebooks, jobs, clusters and large-scale data processing.
• Programming: advanced Python, PySpark and SQL skills.
• ETL & Data Integration: proven experience designing and implementing ETL pipelines and integrating multiple data sources.
• Azure Data Storage: experience with Azure SQL Database, Blob Storage and Azure Data Lake Storage.
• Data Engineering: strong understanding of data warehousing, data modelling and scalable data architectures.
• Ability to develop resilient, modular and fault-tolerant data solutions.
• Strong communication and collaboration across technical teams.

Responsibilities

• Design, develop, test and maintain robust data ingestion pipelines and processes.
• Build scalable, modular and fault-tolerant data solutions using Azure technologies.
• Develop and optimise pipelines using Azure Data Factory and Azure Databricks.
• Work with stakeholders and engineering teams to identify opportunities to improve data products and services.
• Ensure solutions are highly available, resilient, maintainable and traceable.
• Research and implement new approaches to improve data processing and ingestion.
• Collaborate across teams to support smooth data ingestion, consumption and activation.

Skills

8+ years experience

Advanced Python

PySpark

SQL

Strong communication

Collaboration across teams

Tools

Azure Data Factory

Azure Databricks

Apache Spark

Azure SQL Database

Azure Blob Storage

Azure Data Lake Storage

Job description

We are looking for an experienced Data Engineer at Manager level to design, develop and optimise modern data ingestion and processing solutions. You will work collaboratively across teams to build reliable, scalable and traceable data pipelines, while continuously improving data products and services.

Key responsibilities

• Design, develop, test and maintain robust data ingestion pipelines and processes.
• Build scalable, modular and fault-tolerant data solutions using Azure technologies.
• Develop and optimise pipelines using Azure Data Factory and Azure Databricks.
• Work with stakeholders and engineering teams to identify opportunities to improve data products and services.
• Ensure solutions are highly available, resilient, maintainable and traceable.
• Research and implement new approaches to improve data processing and ingestion.
• Collaborate across teams to support smooth data ingestion, consumption and activation.

Essential skills & experience

• 8+ years' experience in data engineering, with experience operating at Manager/Senior level.
• Azure Data Factory: Strong hands-on experience developing and managing pipelines, activities, triggers and data flows.
• Azure Databricks & Apache Spark: Strong experience with notebooks, jobs, clusters and large-scale data processing.
• Programming: Advanced Python, PySpark and SQL skills.
• ETL & Data Integration: Proven experience designing and implementing ETL pipelines and integrating multiple data sources.
• Azure Data Storage: Practical experience with Azure SQL Database, Blob Storage and Azure Data Lake Storage.
• Data Engineering: Strong understanding of data warehousing, data modelling and scalable data architectures.
• Ability to develop resilient, modular and fault-tolerant data solutions.
• Strong communication and collaboration skills, with the ability to work effectively across technical teams.

Preferred skills

• Experience with data pipeline optimisation, performance tuning and cost optimisation.
• Knowledge of data governance, lineage, traceability and data quality practices.
• Experience researching and introducing new technologies or engineering approaches.
• Previous experience leading technical discussions, mentoring engineers or coordinating delivery across teams.