Senior Data Engineer (Databricks & AI-Enabled Data Platforms) (4811)

Keep Simple; São Paulo; Hybrid; BRL 180,000 - 320,000; Full time

Benefits offered by this job

Hybrid work model

Office no escritório de Pinheiros

Job summary

Keep Simple Brasil procura um engenheiro de dados sênior para projetar e otimizar plataformas de dados modernas que suportem analytics, IA/ML, BI e governança. O papel exige liderança técnica, experiência com Databricks, Delta Lake e arquitetura em nuvem, com colaboração entre equipes de engenharia, dados e segurança.

Você atuará na implementação de pipelines, padrões de engenharia, CI/CD, observabilidade e governança de dados, com foco em desempenho, custo e confiabilidade, contribuindo para

Qualifications

• Formação superior em área de tecnologia.
• Experiência com plataformas de dados empresariais em escala.

Responsibilities

• Projetar, construir e manter pipelines de dados escaláveis com Databricks e Spark.
• Conduzir governança de dados, qualidade e observabilidade.
• Conduzir migrações para a nuvem e modernização de dados.

Skills

Databricks

Apache Spark

Delta Lake

Data Modeling

Python

SQL

Azure

AWS

Terraform

Education

Bacharel em Ciência da Computação / TI / Engenharia

Mestrado (preferível)

Tools

Databricks Workflows

Delta Live Tables

Unity Catalog

Job description

Job description

Come work for a large global financial and insurance products company! This is your chance !!

Start a successful career in a renowned company in the international market! Great opportunity!

Global insurance and asset management company seeks a responsible, organized, dynamic and team-oriented person.

Main responsibilities

We are seeking a highly skilled Senior Data Engineer to design, build, and optimize modern data platforms thatenable advanced analytics, AI/ML, and business intelligence solutions. The ideal candidate will possess deepexpertise in Databricks, cloud-based data engineering, data architecture, and modern software engineeringpractices.

This role will provide technical leadership, establish engineering standards, and drive thedevelopment of scalable, secure, and reliable data solutions supporting enterprise-wide initiatives.

Key Responsibilities

Data Platform Engineering:

• Design, develop, and maintain scalable data pipelines using Databricks, Spark, and cloud-native technologies;
• Build and optimize batch and real-time data ingestion frameworks;
• Develop reusable engineering patterns, frameworks, and accelerators for enterprise data solutions;
• Implement data quality, observability, lineage, and governance capabilities across the data ecosystem;
• Support large-scale data modernization and cloud migration initiatives.

Databricks Engineering:

• Design and implement Delta Lake architectures and Medallion data models;
• Build optimized ETL/ELT pipelines using Databricks notebooks, workflows, and Delta Live Tables;
• Establish performance tuning, cost optimization, and workload management best practices;
• Leverage Unity Catalog for data governance, security, and metadata management;
• Implement CI/CD pipelines and Infrastructure-as-Code for Databricks deployments.

Cloud Data Engineering:

• Design cloud-native data architectures across Azure and AWS environments;
• Integrate enterprise data sources including SQL Server, Oracle, APIs, SaaS platforms, and streamingplatforms;
• Build resilient and secure data services supporting high availability and disaster recovery requirements;
• Partner with infrastructure and security teams to implement cloud security controls and compliancestandards.

AI and Advanced Analytics Enablement:

• Build and optimize data products supporting AI, machine learning, and generative AI workloads;
• Develop feature engineering pipelines and curated datasets for model training and inference;
• Support vector databases, semantic search, RAG (Retrieval-Augmented Generation), and LLM-powered applications;
• Collaborate with Data Scientists and AI Engineers to operationalize machine learning solutions;
• Implement MLOps and DataOps practices to support scalable AI delivery.

Technical Leadership:

• Serve as a technical lead for complex data engineering initiatives;
• Define architecture standards, engineering best practices, and development guidelines;
• Mentor junior and mid-level engineers;
• Conduct architecture reviews and provide technical recommendations;
• Drive adoption of automation, observability, and platform engineering principles.

Requirements and skills

Required Qualifications

Education:

• Bachelor's degree in Computer Science, Information Technology, Engineering, or related discipline;
• Master's degree preferred.

Experience

• 8+ years of experience in data engineering, data warehousing, or data platform development;
• 5+ years of hands-on experience with Databricks and Apache Spark;
• Experience designing and implementing enterprise-scale cloud data platforms;
• Experience leading technical initiatives and mentoring engineering teams.

Technical Skills

Data Engineering:

• Databricks;
• Apache Spark (PySpark, Spark SQL);
• Delta Lake;
• Delta Live Tables;
• Unity Catalog;
• Data Warehousing and Data Modeling.

Programming:

• Python;
• SQL;
• Scala (preferred);
• PowerShell or scripting automation experience.

Cloud Platforms:

• Microsoft Azure;
• AWS;
• Cloud-native storage and compute services.

Databases:

• SQL Server;
• Oracle;
• PostgreSQL;
• Snowflake (preferred).

Data Integration:

• Azure Data Factory;
• Databricks Workflows;
• Kafka/Event Streaming;
• REST APIs.

DevOps & Automation:

• GitHub;
• Azure DevOps;
• CI/CD pipelines;
• Terraform;
• Infrastructure as Code.

Observability:

• Monitoring and alerting solutions;
• Data quality frameworks;
• Performance optimization;
• Logging and troubleshooting.

AI & Emerging Technologies:

• Machine Learning data pipelines;
• Feature Stores;
• MLOps;
• Generative AI concepts;
• Vector databases;
• Retrieval-Augmented Generation (RAG);
• AI governance and responsible AI principles.

Preferred Qualifications

• Databricks Certified Data Engineer Professional;
• Azure Data Engineer Associate certification;
• AWS Data Analytics certification;
• Experience supporting regulated industries such as Insurance, Financial Services, or Healthcare;
• Experience implementing enterprise data governance programs;
• Experience integrating AI solutions into production environments.

Key Competencies

• Strategic thinking and problem solving;
• Technical leadership;
• Architecture design;
• Stakeholder management;
• Communication and presentation skills;
• Continuous improvement mindset;
• lnnovation and AI adoption;
• Risk and compliance awareness.

Success Measures

• Delivery of scalable, resilient, and secure data solutions;
• Adoption of enterprise engineering standards and best practices;
• Improvement in platform performance, reliability, and cost efficiency;
• Accelerated delivery of analytics and AI capabilities;
• Reduction in operational support effort through automation and observability;
• Successful mentoring and development of engineering talent.

Additional information

Modelo de contratação:

• PJ.

Forma de atuação:

• Híbrido(3x por semana presencial no escritório de Pinheiros/SP).