Data Engineer

TESTQ Technologies; Stockholms kommun; In loco; SEK 800.000 - 1.100.000; Tempo pieno

Descrizione del lavoro

TESTQ Technologies is seeking an experienced data engineer to design and implement end-to-end data solutions on Azure. You will collaborate with cross-functional teams to deliver scalable analytics, governance, and optimized data pipelines.

The role requires strong Python and SQL skills, hands-on experience with Azure Databricks and Unity Catalog, and a track record of delivering robust CI/CD pipelines in a cloud environment.

Competenze

• 5+ years of hands-on experience delivering end-to-end data solutions using Azure technologies.
• Azure Databricks with Unity Catalog experience for data governance and lineage.

Mansioni

• Design and implement end-to-end data pipelines using CDI mappings and cloud-native architectures.
• Collaborate with cross-functional teams to deliver scalable analytics solutions.
• Lead data governance and pipeline reliability through CI/CD and version control.

Conoscenze

Presentation skills

Cross-functional collaboration

Team collaboration

Problem solving

Independence / self-management

Adaptability

Python

Pandas

NumPy

PySpark

SQL

Git

Azure DevOps

CI/CD

Azure Databricks

Unity Catalog

Azure SQL Database

Synapse Analytics

Data Lake

Strumenti

Azure Databricks

Unity Catalog

Azure SQL Database

Synapse Analytics

Data Lake

Git

CI/CD Pipelines

Azure DevOps

Descrizione del lavoro

• Presentation skills and the ability to adapt communication to the target group
• Very good communication and presentation skills with proven ability to collaborate with cross-functional teams

Leadership and collaboration skills

• Ability to collaborate and work effectively in teams

Problem solving and analysis

• Critical thinking and ability to identify core problems
• Independence and ability to work with little supervision
• Flexibility and adaptability to new situations

Technical skills

Azure Databricks with Unity Catalog

• Expertise in Azure Databricks: Extensive experience with Azure Databricks for data engineering, data science, and analytics solutions.
• Unity Catalog: Proficiency in using Unity Catalog for comprehensive data governance, access control, and lineage tracking across multiple environments.

Advanced Python Skills

• Advanced proficiency in Python for data engineering tasks, including designing and implementing ETL/ELT pipelines and developing reusable, modular code for efficient data transformations.
• Familiarity with Python libraries like Pandas, NumPy, PySpark,

SQL Expertise

• Strong SQL skills to design complex queries and optimize database performance.
• Experience working with Azure databases like Azure SQL Database, Synapse Analytics, and Data Lake for querying, processing, and managing large datasets.

IDMC Data Integration

• Mapping design in Cloud Data Integration (CDI) — Source, Target, Expression, Aggregator, Joiner, Lookup, Filter, Router, Sorter, Rank transformations
• Mapping Task and Task Flow configuration and scheduling
• Parameterization (in/out parameters, parameter files) for reusable, environment-agnostic pipelines
• Working with Mapplets and reusable transformations
• Runtime environments — Secure Agent setup, agent groups, and troubleshooting agent connectivity

Git and CI/CD Expertise

• Version Control: Advanced experience with Git for managing code repositories, versioning, and collaborating in multi-developer environments.
• CI/CD Pipelines: Skilled in designing and implementing CI/CD pipelines using tools like Azure DevOps, GitHub Actions.
• In-depth knowledge of Azure Cloud architecture, including services like Azure Data Lake and Blob Storage.
• Hands-on experience in designing and deploying scalable, cloud-native data pipelines for large datasets.

Experience Summary

• 5+ years of hands‑on experience in delivering end-to-end data solutions using Azure technologies.
• Expertise in Azure Databricks with Unity Catalog for large-scale data processing and governance.
• Advanced skills in Python and SQL for efficient data transformation and analysis.
• Strong experience CI/CD pipelines, and working in Git-based environments for efficient deployment.
• Driving best practices in code quality, data architecture, and DevOps.