Databricks Data Engineer | Senior

Jobgether; Brasil; On-site; BRL 300,000 - 420,000; Full time

Benefits offered by this job

Food and meal allowance

Healthcare: medical and dental

Remote work support: home office

Onboarding kit

Life insurance

Mental health and wellness

Fitness benefits

Professional development

Learning partnerships

Digital content access

Maternity support

Extended parental leave

Employee referral program

Job summary

Jobgether is seeking a Senior Databricks Data Engineer based in Brazil to design and optimize scalable data pipelines using the Databricks ecosystem. You will implement Lakehouse architectures, Medallion patterns, and ensure data governance and security across production environments.

You will mentor colleagues, define best practices, and contribute to cost-efficient, high-performance data platforms leveraging PySpark, Delta Lake, and Azure services.

Qualifications

• Extensive production-grade Databricks experience.
• Proficient with PySpark, Spark SQL, Delta Lake, Apache Spark.
• Hands-on with DLT, CDC, and Structured Streaming.
• Strong knowledge of Unity Catalog and data governance.
• Experience with Azure Data Factory for ingestion and orchestration.
• Proficient in Python for REST APIs and microservices.
• Familiarity with Databricks Workflows, CI/CD, and DevOps practices.

Responsibilities

• Design, develop, and optimize Databricks-based data pipelines.
• Implement lakehouse architectures (Bronze/Silver/Gold).
• Build data ingestion, processing, and integration solutions (CDC, Structured Streaming).
• Manage data governance, security, lineage, and access with Unity Catalog.
• Optimize performance, tuning, and cost efficiency; monitor production pipelines.
• Mentor team members and establish engineering best practices.

Skills

PySpark

Delta Lake

Spark SQL

Azure Data Factory

Databricks Lakehouse

Python

Data governance

Unity Catalog

Performance tuning

Tools

Delta Live Tables (DLT)

Auto Loader

Photon optimization

Structured Streaming

Git CI/CD

Job description

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Senior Databricks Data Engineer based in Brazil.

We are seeking an experienced Senior Databricks Data Engineer to design, develop, and optimize scalable, high-performance data engineering solutions. In this role, you will leverage the Databricks ecosystem to build modern Lakehouse architectures and implement Medallion data models (Bronze, Silver, and Gold). You will be responsible for developing robust data pipelines, integrating multiple data sources, and enabling efficient data processing through technologies such as PySpark, Spark SQL, and Delta Lake. Working with Azure cloud services, you will ensure data quality, governance, security, and operational reliability across production environments. As a technical reference for the team, you will guide engineering best practices, support architectural decisions, and mentor other professionals. This is an opportunity to contribute to advanced data platforms while driving innovation, performance, and cost efficiency in a collaborative technology environment.

Accountabilities

• Design, develop, implement, and continuously improve data engineering solutions using the Databricks ecosystem.
• Define and implement scalable data architectures based on Lakehouse principles and the Medallion architecture (Bronze, Silver, and Gold layers).
• Develop, optimize, and maintain data pipelines using Databricks, PySpark, Spark SQL, Apache Spark, and Delta Lake.
• Implement data ingestion, processing, and integration solutions, including Change Data Capture (CDC) and Structured Streaming.
• Build and manage data pipeline orchestration and system integrations using Azure Data Factory (ADF).
• Develop REST APIs, microservices, and automation solutions using Python.
• Establish and maintain data governance, security, lineage, and access control standards through Unity Catalog.
• Optimize data processing performance, scalability, and infrastructure costs through efficient resource allocation and workload management.
• Apply advanced performance tuning techniques, including Photon optimization, partitioning, shuffle management, and execution plan analysis.
• Implement and maintain observability, monitoring, and operational support practices for production data pipelines.
• Manage Databricks Workflows, CI/CD processes, Git repositories, and Declarative Automation Bundles.
• Identify and resolve complex technical challenges while making architectural decisions aligned with business and engineering requirements.
• Act as a technical reference for the data engineering team, mentoring professionals, sharing knowledge, and establishing engineering best practices.

Requirements

• Extensive professional experience delivering and maintaining production-grade data engineering solutions using Databricks.
• Advanced proficiency in PySpark, Spark SQL, Delta Lake, and Apache Spark.
• Hands-on experience with Delta Live Tables (DLT) / Lakeflow Spark Declarative Pipelines, Change Data Capture (CDC), and Structured Streaming.
• Strong knowledge of Liquid Clustering, Auto Loader, and Delta table optimization techniques.
• Advanced understanding of Unity Catalog, including data governance, security, lineage, and access management.
• Proven experience designing and implementing scalable data architectures, particularly Lakehouse and Medallion architectures.
• Practical experience using Azure Data Factory (ADF) for data ingestion, orchestration, and system integration.
• Advanced Python programming skills, including the development of REST APIs, microservices, and automation solutions.
• Strong expertise in Spark Performance Tuning, including Photon, partitioning strategies, shuffle optimization, and execution plan analysis.
• Experience with Databricks Workflows, CI/CD pipelines, Git, and Declarative Automation Bundles.
• Knowledge of workload optimization, computational resource sizing, and cloud cost reduction strategies.
• Experience with observability, monitoring, troubleshooting, and production pipeline support.
• Ability to independently solve complex technical problems and make sound architectural decisions.
• Previous experience acting as a technical lead or reference, supporting, mentoring, and guiding other engineering professionals.
• Strong analytical thinking, problem-solving skills, autonomy, and effective communication and collaboration abilities.

Required And Preferred Certifications

• Mandatory: Databricks Certified Data Engineer Associate.
• Strongly preferred: Databricks Certified Data Engineer Professional.
• Nice to have: Databricks Certified Associate Developer for Apache Spark.
• Additional certifications in AWS, Microsoft Azure, or Google Cloud Platform (GCP), particularly in Data Engineering and Solutions Architecture, are considered an advantage.

Benefits

• Food and meal allowance: Food and/or meal vouchers to support daily expenses.
• Healthcare: Medical and dental insurance coverage.
• Pet healthcare: Health insurance for your pet.
• Remote work support: Home office allowance for remote positions.
• Welcome kit: Onboarding kit to help you get started in your new role.
• Life insurance: Additional financial protection and security.
• Mental health and wellness: Access to Zenklub, including psychological and nutritional support.
• Fitness benefits: TotalPass membership for physical activities and wellness.
• Professional development: Continuous training and certification programs with leading technology partners, including Adobe, Anthropic, AWS, Google, Microsoft, and Salesforce.
• Learning and development partnerships: Access to exclusive partnerships, discounts, and resources to support your professional growth.
• Digital content: Access to UOL content and resources.
• Maternity support: Special maternity kit.
• Extended parental leave: Enhanced maternity and paternity leave.
• Employee referral program: Opportunities to recommend talented professionals through an internal referral initiative.

Data Privacy Notice:

By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.

We may use artificial intelligence (AI) tools to support parts of the hiring process, analyzing resumes, or assessing responses and identifying potential inconsistencies or verification signals in application materials based on available information. These tools assist our recruitment team but do not replace human judgment. Final hiring decisions are ultimately made by humans. If you would like more information about how your data is processed, please contact us.