Data Engineer

Virtusa Corporation; Stockholms kommun; In loco; SEK 800.000 - 1.100.000; Tempo pieno

Descrizione del lavoro

Virtusa Corporation seeks a senior data engineer to architect and deliver cloud-native data solutions on Azure. You will leverage Azure Databricks, Unity Catalog, and Python for ETL/ELT, while ensuring governance, lineage, and scalable analytics across environments.

The role requires strong SQL skills, experience with Synapse Analytics and data lake storage, and hands-on CI/CD with modern tooling. Collaboration with cross-functional teams is essential for success.

Competenze

• 5+ years of hands-on experience delivering end-to-end data solutions on Azure.
• Expertise with Azure Databricks and Unity Catalog for governance and large-scale processing.
• Advanced Python and SQL skills for ETL/ELT and data analysis.
• Solid CI/CD experience using Azure DevOps or GitHub Actions.
• Experience building cloud-native data pipelines and data integration workflows.

Mansioni

• Design and implement end-to-end data pipelines on Azure.
• Set up and maintain CI/CD pipelines for data projects.
• Govern data access and lineage with Unity Catalog.
• Optimize SQL queries and data lake architectures on Azure.
• Collaborate with data science and analytics teams to deliver scalable solutions.

Conoscenze

Azure Databricks

Unity Catalog

Python

PySpark

Pandas

NumPy

SQL

Azure SQL

Synapse Analytics

Strumenti

Git

Azure DevOps

GitHub Actions

Azure Data Lake

Blob Storage

Descrizione del lavoro

• Expertise in Azure Databricks: Extensive experience with Azure Databricks for data engineering, data science, and analytics solutions.


• Unity Catalog: Proficiency in using Unity Catalog for comprehensive data governance, access control, and lineage tracking across multiple environments.

Advanced Python Skills

• Advanced proficiency in Python for data engineering tasks, including designing and implementing ETL/ELT pipelines and developing reusable, modular code for efficient data transformations.


• Familiarity with Python libraries like Pandas, NumPy, PySpark,

SQL Expertise

• Strong SQL skills to design complex queries and optimize database performance.


• Experience working with Azure databases like Azure SQL Database, Synapse Analytics, and Data Lake for querying, processing, and managing large datasets.

IDMC Data Integration

• Mapping design in Cloud Data Integration (CDI) — Source, Target, Expression, Aggregator, Joiner, Lookup, Filter, Router, Sorter, Rank transformations


• Mapping Task and Task Flow configuration and scheduling


• Parameterization (in/out parameters, parameter files) for reusable, environment-agnostic pipelines


• Working with Mapplets and reusable transformations


• Runtime environments — Secure Agent setup, agent groups, and troubleshooting agent connectivity

Git and CI/CD Expertise

• Version Control: Advanced experience with Git for managing code repositories, versioning, and collaborating in multi-developer environments.


• CI/CD Pipelines: Skilled in designing and implementing CI/CD pipelines using tools like Azure DevOps, GitHub Actions.


• In-depth knowledge of Azure Cloud architecture, including services like Azure Data Lake and Blob Storage.


• Hands-on experience in designing and deploying scalable, cloud-native data pipelines for large datasets.

Experience Summary:

• 5+ years of hands-on experience in delivering end-to-end data solutions using Azure technologies.


• Expertise in Azure Databricks with Unity Catalog for large-scale data processing and governance.


• Advanced skills in Python and SQL for efficient data transformation and analysis.


• Strong experience CI/CD pipelines, and working in Git-based environments for efficient deployment.


• Driving best practices in code quality, data architecture, and DevOps.