Senior Data Engineer (Databricks & Ai-Enabled Dataplatforms) (4811)

Keep Simple; São Paulo; Remote; BRL 180,000 - 300,000; Full time

Job summary

A Keep Simple, empresa de consultoria em TI com foco na experiência do candidato, busca Senior Data Engineer para projetar e otimizar plataformas de dados modernas que habilitem analytics, IA/ML e BI.

O candidato ideal possui experiência aprofundada em Databricks, arquitetura de dados em nuvem e governança de dados, atuando com liderança técnica e padrões de engenharia em equipes de dados.

Qualifications

• Bacharelado em Ciência da Computação, TI, Engenharia ou área relacionada.

Responsibilities

• Desenhar, construir e manter pipelines de dados escaláveis usando Databricks, Spark e tecnologias em nuvem.
• Implementar qualidade de dados, observabilidade, lineage e governança no ecossistema de dados.
• Apoiar modernização de dados em nube e migrações para a cloud.
• Desenvolver padrões de engenharia, automatizar pipelines e práticas CI/CD.
• Atuar como liderança técnica, orientando engenheiros e promovendo melhores práticas.

Skills

Databricks

Apache Spark

Python

SQL

Scala

Azure

AWS

Delta Lake

Data Warehousing

Data Modeling

GitHub

CI/CD

Education

Bacharelado em Ciência da Computação, TI, Engenharia ou área relacionada

Mestrado preferencial

Tools

Databricks

Delta Live Tables

Unity Catalog

Azure Data Factory

Kafka

REST APIs

Job description

Somos uma empresa de consultoria em TI com mais de 10 anos no mercado e contamos com um time de especialistas em recrutamento tech. Nosso processo é 100% focado na experiência de quem tanto importa, o candidato.

Optamos por fazer a diferença e temos orgulho em dizer que todos que passam pela Keep Simple se sentem especiais. Possuímos um ambiente descontraído, colaborativo, e adotamos o ágil de verdade.

Faça parte da nossa história, #vemprakeep

Required Qualifications

Education:

• Bachelor's degree in Computer Science, Information Technology, Engineering, or related discipline;
• Master's degree preferred.

Experience

• 8+ years of experience in data engineering, data warehousing, or data platform development;
• 5+ years of hands‑on experience with Databricks and Apache Spark;
• Experience designing and implementing enterprise‑scale cloud data platforms;
• Experience leading technical initiatives and mentoring engineering teams.

Technical Skills

Data Engineering:

• Databricks;
• Apache Spark (PySpark, Spark SQL);
• Delta Lake;
• Delta Live Tables;
• Unity Catalog;
• Data Warehousing and Data Modeling.

Programming:

• Python;
• SQL;
• Scala (preferred);
• PowerShell or scripting automation experience.

Cloud Platforms:

• Microsoft Azure;
• AWS;
• Cloud‑native storage and compute services.

Databases:

• SQL Server;
• Oracle;
• PostgreSQL;
• Snowflake (preferred).

Data Integration:

• Azure Data Factory;
• Databricks Workflows;
• Kafka/Event Streaming;
• REST APIs.

DevOps & Automation:

• GitHub;
• Azure DevOps;
• CI/CD pipelines;
• Terraform;
• Infrastructure as Code.

Observability:

• Monitoring and alerting solutions;
• Data quality frameworks;
• Performance optimization;
• Logging and troubleshooting.

AI & Emerging Technologies:

• Machine Learning data pipelines;
• Feature Stores;
• MLOps;
• Generative AI concepts;
• Vector databases;
• Retrieval‑Augmented Generation (RAG);
• AI governance and responsible AI principles.

Preferred Qualifications

• Databricks Certified Data Engineer Professional;
• Azure Data Engineer Associate certification;
• AWS Data Analytics certification;
• Experience supporting regulated industries such as Insurance, Financial Services, or Healthcare;
• Experience implementing enterprise data governance programs;
• Experience integrating AI solutions into production environments.

Key Competencies

• Strategic thinking and problem solving;
• Technical leadership;
• Architecture design;
• Stakeholder management;
• Communication and presentation skills;
• Continuous improvement mindset;
• lnnovation and AI adoption;
• Risk and compliance awareness.

Success Measures

• Delivery of scalable, resilient, and secure data solutions;
• Adoption of enterprise engineering standards and best practices;
• Improvement in platform performance, reliability, and cost efficiency;
• Accelerated delivery of analytics and AI capabilities;
• Reduction in operational support effort through automation and observability;
• Successful mentoring and development of engineering talent;
• Fluent english.

We are seeking a highly skilled Senior Data Engineer to design, build, and optimize modern data platforms that enable advanced analytics, AI/ML, and business intelligence solutions. The ideal candidate will possess deep expertise in Databricks, cloud‑based data engineering, data architecture, and modern software engineering practices.

This role will provide technical leadership, establish engineering standards, and drive the development of scalable, secure, and reliable data solutions supporting enterprise‑wide initiatives.

Key Responsibilities

Data Platform Engineering:

• Design, develop, and maintain scalable data pipelines using Databricks, Spark, and cloud‑native technologies;
• Build and optimize batch and real‑time data ingestion frameworks;
• Develop reusable engineering patterns, frameworks, and accelerators for enterprise data solutions;
• Implement data quality, observability, lineage, and governance capabilities across the data ecosystem;
• Support large‑scale data modernization and cloud migration initiatives.

Databricks Engineering:

• Design and implement Delta Lake architectures and Medallion data models;
• Build optimized ETL/ELT pipelines using Databricks notebooks, workflows, and Delta Live Tables;
• Establish performance tuning, cost optimization, and workload management best practices;
• Leverage Unity Catalog for data governance, security, and metadata management;
• Implement CI/CD pipelines and Infrastructure‑as‑Code for Databricks deployments.

Cloud Data Engineering:

• Design cloud‑native data architectures across Azure and AWS environments;
• Integrate enterprise data sources including SQL Server, Oracle, APIs, SaaS platforms, and streaming platforms;
• Build resilient and secure data services supporting high availability and disaster recovery requirements;
• Partner with infrastructure and security teams to implement cloud security controls and compliance standards.

AI and Advanced Analytics Enablement:

• Build and optimize data products supporting AI, machine learning, and generative AI workloads;
• Develop feature engineering pipelines and curated datasets for model training and inference;
• Support vector databases, semantic search, RAG (Retrieval‑Augmented Generation), and LLM‑powered applications;
• Collaborate with Data Scientists and AI Engineers to operationalize machine learning solutions;
• Implement MLOps and DataOps practices to support scalable AI delivery.

Technical Leadership:

• Serve as a technical lead for complex data engineering initiatives;
• Define architecture standards, engineering best practices, and development guidelines;
• Mentor junior and mid‑level engineers;
• Conduct architecture reviews and provide technical recommendations;
• Drive adoption of automation, observability, and platform engineering principles.