Senior Data Engineer (Data & AI) | Híbrido: São Paulo, SP

Indicium AI; São Paulo; Remote; BRL 180,000 - 260,000; Full time

Benefits offered by this job

Remote work

Annual bonus

Health insurance

Career development

English classes

Job summary

Indicium AI is seeking a Senior Data Engineer to modernize lakehouses on Azure Databricks, enabling domain data products and AI use cases. You will lead architecture decisions, secure data pipelines, and mentor junior engineers while driving cost reductions and production stability.

You will work on ingestion redesign, Unity Catalog governance, and CI/CD practices in a distributed squad, with exposure to industrial time series data and operational intelligence initiatives.

Qualifications

• Senior data engineer with production experience in Python and PySpark/Spark SQL.
• Proficient with Delta Lake and Databricks, including Unity Catalog.
• Experience in ingestion patterns (CDC, incremental loads) and data security/compliance (LGPD).
• Familiarity with IaC (Terraform) and CI/CD for data platforms.
• Ability to cost-optimize cloud data platforms and mentor junior engineers.

Responsibilities

• Redesign data ingestion to preserve history with raw/landing layer.
• Define lakehouse architecture including DEV/HML/PRD environments and Unity Catalog strategy.
• Protect sensitive data with masking, role-based access, and ABAC in Unity Catalog.
• orchestrate pipelines with Lakeflow, Terraform, and CI/CD on Azure DevOps.
• Ensure reliability with alerts, SLAs, data quality, and observability.
• Lead FinOps decisions and mentor junior engineers; handover to support.
• Support ingestion of industrial time series for operational intelligence.
• Review code, write ADRs, and promote architectural decisions.

Skills

Python

PySpark/Spark SQL

Delta Lake

Databricks

Unity Catalog

CDC

Ingestion patterns

LGPD compliance

Data security

Cloud cost optimization

Tools

Terraform

Azure DevOps

Lakeflow

Databricks Asset Bundles

SQL Warehouse

Job description

About Indicium AI

Indicium AI is trusted by the world's leading enterprises to deliver AI into production at scale. We are a global, AI-native consultancy with deep expertise across Financial Services, Energy & Utilities, Healthcare & Life Sciences, Retail & CPG, and Manufacturing - guiding organizations from strategy through build to measurable business outcomes. With 600+ AI experts, 50+ enterprise clients, and five global locations, we work side-by-side with the world's leading AI partners - including Anthropic, Databricks, AWS, OpenAI, and Microsoft - to deliver modern AI with speed, clarity, and lasting impact.

As a global company, we offer career opportunities across Europe and the Americas and are looking for talent eager to grow globally with our team. Please indicate in your application if you are available to work abroad and if you hold a European passport.

About The Opportunity

The Senior Data Engineer will act as the technical reference for data platforms on projects for industrial clients, modernizing lakehouses on Azure Databricks so they can support domain-based data products and AI use cases in operations. The role spans from diagnosing legacy platforms to designing the architecture for ingestion, environments, security, and cost control in production.

The position involves redesigning ingestion patterns with history preservation, separating development and production environments, protecting sensitive data in compliance with the LGPD (Brazilian General Data Protection Law), and reducing costs measurably. It also includes leading platform decisions within the squad, documenting them, and mentoring mid-level and junior data engineers, as well as preparing the handover to the support team.

We are looking for a hands‑on professional who writes code, defends architecture decisions with data, and is driven to see stable, secure, and efficient platforms running in real‑world operations.

Key Responsibilities

• Redesigning ingestion with a raw/landing layer that preserves history, replacing full loads with overwrite by incremental and CDC patterns from SQL Server staging and source systems (ERP, HR, maintenance, SSMA, and operational systems).
• Defining the lakehouse architecture: medallion layers, separate DEV/HML/PRD environments, and a domain-based Unity Catalog strategy.
• Implementing data security and privacy by removing credentials from code (Key Vault, secret scopes, and service principals) and applying column masks, row filters, and ABAC in Unity Catalog to classified personal data, in compliance with the LGPD.
• Orchestrating pipelines with Lakeflow Jobs and Databricks Asset Bundles, using infrastructure as code (Terraform) and CI/CD on Azure DevOps, including linting, testing, and promotion across environments.
• Ensuring platform reliability through alerts with clear ownership and escalation, data refresh SLAs, data quality expectations, and observability using Databricks system tables.
• Leading platform FinOps (serverless vs. classic compute, cluster policies, budgets, and tags), with measurable cost reduction.
• Taking over and stabilizing legacy pipelines, leading incident handling, creating runbooks, and transferring knowledge to the support team.
• Supporting, when needed, the ingestion of industrial time series (process historian) for operational intelligence products.
• Reviewing code, writing ADRs, and mentoring mid-level and junior data engineers.
• Having the freedom and critical thinking to propose and challenge data engineering solutions.

Preferred Qualifications & Experience

• Senior experience in production data engineering with Python, PySpark/Spark SQL, and Delta Lake.
• Production experience with Databricks: Unity Catalog, Jobs/Lakeflow, Databricks Asset Bundles, and SQL Warehouse.
• Strong command of ingestion patterns: CDC, incremental loads, Auto Loader, JDBC, REST APIs, and schema evolution.
• Experience with infrastructure as code (Terraform) and CI/CD for data platforms.
• Hands‑on experience with data security and privacy: personal data masking, access models, and LGPD.
• Track record of reducing costs on cloud data platforms.
• Experience taking over and refactoring legacy data platforms.
• Ability to communicate technical and cost trade‑offs to technical leadership and to defend architecture decisions against third‑party recommendations.
• Autonomy to deliver at a fast pace in legacy environments, without a long learning curve on the stack.
• Fluent Portuguese and intermediate/advanced English.

Nice to haves

• Experience with Azure data services: ADLS Gen2, Key Vault, Entra ID, and networking fundamentals (Private Link, VNet).
• SAP data integration (S/4HANA, SAP Datasphere / Business Data Cloud, Delta Sharing) and TOTVS Protheus.
• Experience with industrial data: AVEVA PI System / PI Web API, OPC UA, MQTT, Structured Streaming, or Azure Event Hubs.
• Knowledge of dbt.
• Experience migrating legacy BI (Qlik to Power BI).
• Experience with SOX / ITGC controls on data platforms.
• Databricks Data Engineer Professional or Azure certification.
• Experience with heavy industry data.

Why Indicium AI

• Work on AI projects that actually transform the world's largest enterprises
• Use cutting‑edge AI tools and technologies every single day
• Collaborate with global teams on high‑impact, real‑world solutions
• Be backed by a supportive team that's genuinely in your corner
• Benefit from serious investment in your learning and career growth
• Earn competitive compensation and benefits
• Enjoy company events and gatherings that bring the global team together
• Join a fast‑growing company where ambitious careers thrive

Selection Process Stages

• Interview with Talent Acquisition
• Technical Challenge
• Interview with Technical Management
• Evidence Storage

Some of the benefits offered by Indicium:

Our Culture

• Modern, transparent, and inclusive management.
• Dynamic environment, open to innovation and creativity.
• Data‑driven culture.
• Autonomy and engagement.
• Light and relaxed atmosphere with a focus on challenges.
• High‑growth environment (Fast‑paced scaling).
• Remote work and flexible hours.

For Your Professional Development

• Structured career path.
• Feedback and growth culture (45/90‑day onboarding reviews and semi‑annual performance cycles).
• Mentorship Program for dedicated support during onboarding.
• Experienced and collaborative team.
• High‑impact projects.
• Agile methodologies and cross‑functional teams.
• Free weekly English conversation classes.
• Certification Program (full financial support and performance rewards).
• Tutoring opportunities at Indicium Academy.
• Internal initiatives such as "Tech Tracks" and "Databloggers".

For Your Health and Wellbeing

• Global offices in Florianópolis, São Paulo, New York, Lisbon, and London.
• In‑office snacks and beverages.
• Points and rewards program.
• Psychological support via "Programa Oxigênio" and Zenklub.
• Birthday day off.
• Weekly happy hours and game nights (both remote and in‑person).
• Wellhub (formerly Gympass).
• Bradesco Health Insurance.
• Flash Card (flexible benefits - total R$ 1,350.00).
• Annual bonus (performance‑based).
• Referral Program (R$ 250 per successful hire).
• Extended maternity leave (6 months).

About The Company

Indicium AI is trusted by the world's leading enterprises to deliver AI into production at scale. We are a global AI‑native consultancy with proven experience across Financial Services, Energy & Utilities, Healthcare & Life Sciences, Retail & CPG, and Manufacturing. From strategy, to build, to business outcomes, we unlock value from AI with unmatched clarity, speed, and capability.

Powered by 600+ AI experts serving 50+ enterprise clients from 5 global locations, we work side‑by‑side with top partners - including Anthropic, Databricks, AWS, OpenAI, and Microsoft - to deliver modern AI with speed and measurable impact.