Senior Data Engineer (Databricks & AI Platforms) (4809)

Keep Simple; São Bernardo do Campo; Hybrid; BRL 240,000 - 360,000; Full time

Benefits offered by this job

Hybrid work

Office Pinheiros

Job summary

Keep Simple is seeking a Senior Data Engineer to design, build, and optimize scalable data platforms and pipelines for analytics, ML, and Generative AI initiatives in a hybrid setup in São Paulo, Brazil.

The role partners with data scientists and AI engineers to deliver trusted data products using Databricks and cloud technologies, while ensuring governance, security, and observability across data platforms.

Responsibilities

• Design, develop, and maintain scalable data pipelines and data products using Databricks, Spark, Python, and SQL.
• Build and optimize batch, streaming, and real-time data integration solutions from enterprise and third-party data sources.
• Implement Databricks Lakehouse architectures utilizing Delta Lake and Medallion design patterns.
• Develop and maintain data products that support analytics, predictive modeling, machine learning, and Generative AI applications.
• Collaborate with Data Scientists and AI Engineers to prepare, transform, and govern data for AI and ML use cases.

Job description

Job description

Come work for a large global financial and insurance products company! This is your chance !!

Start a successful career in a renowned company in the international market! Great opportunity!

Global insurance and asset management company seeks a responsible, organized, dynamic and team-oriented person.

Main responsibilities

The Senior Data Engineer is responsible for designing, building, and optimizing scalable data platforms and pipelines that support analytics, business intelligence, machine learning (ML), and AI-driven solutions.

This role partners with data scientists, AI engineers, architects, and business stakeholders to deliver trusted, high-quality data products using Databricks, cloud technologies, and modern data engineering practices. The position also serves as a technical leader in enabling enterprise AI and Generative AI initiatives through robust, secure, and governed data platforms.

Key Responsibilities:

• Design, develop, and maintain scalable data pipelines and data products using Databricks, Spark, Python, and SQL;
• Build and optimize batch, streaming, and real-time data integration solutions from enterprise and third-party data sources;
• Implement Databricks Lakehouse architectures utilizing Delta Lake and Medallion design patterns;
• Develop and maintain data products that support analytics, predictive modeling, machine learning, and Generative AI applications;
• Collaborate with Data Scientists and AI Engineers to prepare, transform, and govern data for AI and ML use cases;
• Design and implement feature engineering pipelines and support ML lifecycle processes;
• Develop data solutions that support Retrieval Augmented Generation (RAG), vector search, semantic search, and LLM-based applications;
• Optimize Spark jobs, SQL workloads, and data processing frameworks for performance, scalability, and cost efficiency;
• Implement data quality, observability, lineage, governance, and monitoring capabilities;
• Ensure compliance with data privacy, security, and responsible AI standards;
• Contribute to CI/CD, Infrastructure-as-Code, and DataOps practices across the data platform;
• Mentor junior engineers and promote engineering best practices across the organization.

Requirements and skills

Required Qualifications:

• Bachelor's degree in Computer Science, Information Systems, Engineering, Mathematics, or a related field;
• 7+ years of experience in data engineering, ETL development, or large-scale data platform engineering;
• 3+ years of hands‑on experience with Databricks and Apache Spark;
• Strong proficiency in Python, SQL, and distributed data processing frameworks;
• Experience building cloud-based data lakes, data warehouses, and Lakehouse architectures;
• Experience supporting AI, machine learning, or advanced analytics initiatives;
• Strong understanding of data modeling, data governance, and enterprise data management practices;
• Experience developing and optimizing large-scale data pipelines in AWS, Azure, or Google Cloud.

Preferred Qualifications:

• Experience with Databricks Delta Lake, Unity Catalog, Delta Live Tables, MLflow, Mosaic AI, and Databricks Workflows;
• Hands‑on experience supporting Generative AI, Large Language Models (LLMs), RAG architectures, vector databases, or AI-powered applications;
• Familiarity with AI frameworks such as LangChain, Semantic Kernel, OpenAI APIs, Hugging Face, or similar technologies;
• Experience with feature stores, model training pipelines, and machine learning operationalization (MLOps);
• Experience with Kafka, Event Hub, or other streaming technologies;
• Databricks Certified Data Engineer Professional or equivalent cloud certification.

Key Competencies:

• Databricks Platform Engineering;
• Apache Spark Development;
• AI & Machine Learning Data Engineering;
• Generative AI Data Solutions;
• Lakehouse Architecture;
• Data Modeling & Data Warehousing;
• Data Governance & Security;
• Data Observability & Reliability Engineering;
• Cloud Data Platforms (AWS/Azure);
• DataOps, CI/CD & Automation;
• Technical Leadership & Mentoring.

Success Measures:

• Delivery of scalable, reliable, and secure data platforms supporting analytics, AI, and business operations;
• Successful enablement of AI and Generative AI use cases through high-quality, governed data products;
• Improved data quality, pipeline reliability, and platform performance;
• Increased automation, operational efficiency, and reuse of engineering frameworks;
• Adoption of data engineering standards and best practices across development teams;
• Measurable improvements in AI/ML solution delivery speed and business value realization.

Additional information

Modelo de contratação:

• PJ.

Forma de atuação:

• Híbrido(3x por semana presencial no escritório de Pinheiros/SP).