Data Engineer

Luxoft; Brasil; Presencial; BRL 200 000 - 420 000; Tempo integral

Resumo da oferta

Luxoft is seeking a Senior Data Engineer to build a scalable data platform for a major retail client in Brazil. You will focus on processing large data volumes and integrating with existing architecture.

You will design data models, implement ETL/ELT pipelines, and work across teams to ensure reliable data delivery using Azure technologies such as Data Factory, Synapse, and Databricks.

Qualificações

• 8+ years of data engineering experience.
• Hands-on Azure Data Factory and Synapse (3+ years).
• Design and implement data models for storage and retrieval.
• Strong Azure stack knowledge: ADLS, Synapse, Data Factory, Databricks.
• Build robust ETL/ELT pipelines to data lakes/warehouses.
• Integrate data from databases, APIs, and external providers.
• Design data warehousing with dimensional modeling, star schemas, Data Mesh/Delta Lakehouse.
• Proficient in SQL for complex queries and performance tuning in cloud storage.
• Experience with metadata governance in cloud data platforms.
• Azure/Databricks certifications add value.
• Experience with cloud-based analytical databases.
• Experience with Azure MI, Azure Postgres, Cosmos DB, Analysis Services, Informix.
• Python for ETL.
• Shell scripting ( Bash/Unix/Windows ).
• Proven cross-functional leadership, architecture, mentoring, and stakeholder communication.

Responsabilidades

• Develop and implement data platform solutions for large-scale data processing.
• Collaborate with cross-functional teams to drive data strategy.
• Mentor engineers and communicate with stakeholders.

Conhecimentos

Azure Data Factory

Synapse

Data Modeling

Azure Data Lake

Azure Databricks

pySpark

ETL/ELT Pipelines

API Integration

SQL

Metadata Governance

Data Warehousing

Certifications (Azure/Databricks)

Postgres/Azure Postgres

Cosmos DB

Analysis Services

Informix

Python

Shell Scripting

Leadership & Communication

Descrição da oferta de emprego

We are looking for Senior Data Engineer to implement a solution for a big retail company. The main focus is to support processing of big data volumes and integrate solution to current architecture.

Mandatory Skills Description:

• Overall years of experience required 8+
• Strong, recent hands-on expertise with Azure Data Factory and Synapse is a must (3+ years)
• Strong expertise in designing and implementing data models, including conceptual, logical, and physical data models, to support efficient data storage and retrieval
• Strong knowledge of Microsoft Azure, including Azure Data Lake Storage, Azure Synapse Analytics, Azure Data Factory, and Azure Databricks, pySpark for building scalable and reliable data solutions
• Extensive experience with building robust and scalable ETL/ELT pipelines to extract, transform, and load data from various sources into data lakes or data warehouses
• Ability to integrate data from disparate sources, including databases, APIs, and external data providers, using appropriate techniques such as API integration or message queuing
• Proficiency in designing and implementing data warehousing solutions (dimensional modeling, star schemas, Data Mesh, Data/Delta Lakehouse, Data Vault)
• Proficiency in SQL to perform complex queries, data transformations, and performance tuning on cloud-based data storages
• Experience integrating metadata and governance processes into cloud-based data platform
• Certification in Azure, Databricks, or other relevant technologies is an added advantage
• Experience with cloud-based analytical databases
• Experience with Azure MI, Azure Database for Postgres, Azure Cosmos DB, Azure Analysis Services, and Informix
• Experience with Python and Python-based ETL tools
• Experience with shell scripting in Bash, Unix or windows shell is preferable
• Demonstrated ability to lead cross-functional engineering teams, define technical strategy and architecture, drive delivery of complex data platforms, mentor engineers, and effectively communicate with stakeholders at all organizational levels

Nice-to-Have Skills Description

• Experience with Elasticsearch
• Familiarity with containerization and orchestration technologies (Docker, Kubernetes)
• Troubleshooting and Performance Tuning: Ability to identify and resolve performance bottlenecks in data processing workflows and optimize data pipelines for efficient data ingestion and analysis.
• Collaboration and Communication: Strong interpersonal skills to collaborate effectively with stakeholders, data engineers, data scientists, and other cross-functional teams
• Ability to plan, estimate and track progress of implementing features
• Computer Science and data science academic and education credentials