Sr Data Engineer

Luxoft; São Paulo; Presencial; BRL 80.000 - 120.000; Tempo integral

Resumo da oferta

A leading IT consulting firm in Brazil is seeking a skilled Data Engineer who specializes in designing and implementing scalable data pipelines using Databricks and Kafka. The role requires at least 5 years of experience in data engineering, developing real-time streaming solutions, and a strong command of cloud platforms like AWS or Azure. Candidates with proficiency in Python or Scala are preferred. This full-time position offers opportunities to collaborate on cross-functional teams while ensuring data quality and compliance.

Qualificações

• 5 years of experience in data engineering roles.
• Proven expertise with Databricks, including Spark and Delta Lake.
• Strong hands-on experience with Apache Kafka.

Responsabilidades

• Design and implement scalable data pipelines using Databricks and Kafka.
• Build and maintain real-time streaming solutions for high-volume data.
• Collaborate with cross-functional teams to integrate data flows.

Conhecimentos

Data engineering

Databricks

Apache Kafka

Python

CI / CD pipelines

Cloud platforms (AWS, Azure, GCP)

Ferramentas

GitLab

Jenkins

Descrição da oferta de emprego

Responsibilities

Design and implement scalable data pipelines using Databricks and Kafka

Build and maintain real-time streaming solutions for high-volume data

Collaborate with cross-functional teams to integrate data flows into broader systems

Optimize performance and reliability of data processing workflows

Ensure data quality, lineage, and compliance across streaming and batch pipelines

Participate in agile development processes and contribute to technical documentation

Mandatory Skills Description

• 5 years of experience in data engineering roles
• Proven expertise with Databricks (Spark, Delta Lake, notebooks, performance tuning)
• Strong hands‑on experience with Apache Kafka (topics, producers / consumers, schema registry)
• Solid understanding of streaming frameworks (e.g., Spark Structured Streaming, Flink, or similar)
• Experience with cloud platforms (AWS, Azure, or GCP)
• Proficiency in Python or Scala for data pipeline development
• Familiarity with CI / CD pipelines (GitLab, Jenkins) and agile tools (Jira)
• Exposure to data lakehouse architectures and best practices
• Knowledge of data governance, security, and observability

Project Description

We are seeking a skilled and hands‑on Data Engineer with proven experience in Databricks, Apache Kafka, and real‑time data streaming solutions.

Seniority Level

Mid‑Senior level

Employment Type

Full‑time

Job Function

Business Development, Information Technology, and Analyst

Industries

IT Services and IT Consulting

Referrals

Referrals increase your chances of interviewing at Luxoft by 2x