Senior Data Engineer - Data Engineering

GEP

São Paulo

On-site

BRL 350,000 - 520,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

GEP in Brazil is seeking a senior data engineering lead to design and optimize scalable data solutions on Databricks. You will architect Delta Lake lakehouse patterns, govern Unity Catalog data assets, and build robust batch/incremental pipelines with Lakeflow.

You will own CI/CD with DABs and Git, ensure data quality, and mentor engineers while collaborating with analysts and product stakeholders. The role emphasizes strong Python/SQL, Spark tuning, and cloud experience (Azure Databricks

Qualifications

  • Databricks expertise including Lakeflow Jobs and Delta Lake.
  • Strong Spark performance tuning.
  • Expert Python and SQL for data engineering.
  • Unity Catalog governance with lineage.
  • CI/CD for data solutions (DABs, Git, testing).
  • Knowledge of lakehouse and medallion architecture.
  • Data-quality, observability, and data contracts.
  • Azure Databricks preferred; cloud AWS/GCP valuable.
  • Web scraping tools knowledge is a plus.
  • Procurement/foreign trade data knowledge is a plus.
  • Strong communication and mentoring skills.

Responsibilities

  • Lead the design, development, and optimization of scalable data solutions on Databricks using Python, SQL, and PySpark.
  • Architect lakehouse solutions based on Delta Lake and medallion architecture, defining data models, layer boundaries, and performance strategies.
  • Define and govern data assets in Unity Catalog, including catalog and schema structure, access permissions, and lineage.
  • Design and build batch and incremental pipelines with Lakeflow Jobs and Lakeflow Declarative Pipelines, including MERGE/CDC patterns and data-quality expectations.
  • Own CI/CD for our Databricks solutions, packaging and deploying jobs and pipelines across environments with Databricks Asset Bundles (DABs), Git, and automated testing.
  • Establish data-quality, observability, and data-contract practices that ensure the accuracy, consistency, and agreed service levels of business-critical datasets.
  • Optimize Databricks workloads for performance and cost, including compute selection (classic and serverless), Spark tuning, and Delta table maintenance.
  • Provide technical leadership in architectural decisions, code reviews, and the troubleshooting of critical pipeline and platform issues.
  • Mentor junior and mid-level engineers, fostering a culture of learning and technical excellence, and take part in technical interviews and assessments.
  • Collaborate with analysts, product managers, and business stakeholders to translate business needs into data solutions and communicate technical trade-offs.

Skills

Databricks
Spark
Python
SQL
Unity Catalog
CI/CD
Data modeling
Data quality
Azure Databricks
Cloud platforms
AI tools
Web scraping
Procurement data
Agile
Mentorship
Git
Databricks Asset Bundles

Tools

Databricks Asset Bundles (DABs)
Git

Job description

Full Time
Job Description
Welcome to GEP
Who We Are

GEP, with over 30 offices internationally, is the fastestgrowing procurement and supply chain solutions firm - consulting, software andmanaged services. GEP has succeeded by being smart and creative; solvingcomplex problems and finding opportunities for the world’s largestorganizations.

How You Grow at GEP

We recognize people who make a genuine difference, workhard and drive achievements. Results are noticed and rewarded. It’s how youwill grow a career at GEP, and in a much shorter time frame than at otherfirms.

Celebrating Everyone

GEP succeeds through the ideas and creativity of our teammembers so we embrace people of all experiences, nationalities, abilities,cultures, races, gender identities, sexual orientations and ages. What makesyou unique and different is celebrated and will help GEP stand out even more.And we are a women-founded and -owned company so our foundation is making GEP agreat place to work for women, a place where women can learn, advance and giveback.

What You Will Do
  • Lead the design, development, and optimization of scalable data solutions on Databricks using Python, SQL, and PySpark.
  • Architect lakehouse solutions based on Delta Lake and medallion architecture, defining data models, layer boundaries, and performance strategies.
  • Define and govern data assets in Unity Catalog, including catalog and schema structure, access permissions, and lineage.
  • Design and build batch and incremental pipelines with Lakeflow Jobs and Lakeflow Declarative Pipelines, including MERGE/CDC patterns and data-quality expectations.
  • Own CI/CD for our Databricks solutions, packaging and deploying jobs and pipelines across environments with Databricks Asset Bundles (DABs), Git, and automated testing.
  • Establish data-quality, observability, and data-contract practices that ensure the accuracy, consistency, and agreed service levels of business-critical datasets.
  • Optimize Databricks workloads for performance and cost, including compute selection (classic and serverless), Spark tuning, and Delta table maintenance.
  • Provide technical leadership in architectural decisions, code reviews, and the troubleshooting of critical pipeline and platform issues.
  • Mentor junior and mid-level engineers, fostering a culture of learning and technical excellence, and take part in technical interviews and assessments.
  • Collaborate with analysts, product managers, and business stakeholders to translate business needs into data solutions and communicate technical trade-offs.
What Should You Bring
  • Expert-level hands-on experience with Databricks, including Lakeflow Jobs, Lakeflow Declarative Pipelines, Spark DataFrames, and Delta Lake.
  • Deep understanding of Spark and distributed data processing, including partitioning, joins, shuffles, and performance tuning.
  • Expert-level Python and SQL skills for data engineering, including PySpark, Pandas, and writing testable, production-grade code.
  • Strong experience with Unity Catalog governance, including catalog design, access control, and lineage.
  • Strong experience with CI/CD for data solutions, including Databricks Asset Bundles (DABs), Git-based workflows, and automated testing (e.g., pytest).
  • Solid knowledge of data modeling, lakehouse and medallion architecture, incremental processing (MERGE, CDC), and Delta Lake optimization.
  • Knowledgedesigning data-quality frameworks, observability, and data contracts for production pipelines.
  • Expertise with Azure Databricks is strongly preferred. Experience with AWS, Google Cloud, or another cloud platform is also valuable.
  • Familiarity with AI-assisted development tools and Databricks AI Functions is a plus.
  • Kkoeledge with web scraping tools and techniques (e.g., BeautifulSoup, Scrapy, Selenium) is a plus.
  • Knowledge of procurement, public procurement, or foreign trade (import/export) data is a plus.
  • Proven ability to lead projects in agile environments, balancing technical and business priorities.
  • Excellent communication and mentorship skills to guide engineers and present solutions to stakeholders.
Are You One of Us

GEP is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, ethnicity, color, national origin, religion, sex, protected veteran status, disability status, or any other characteristics protected by federal, state or local law. We are committed to hiring and valuing a global diverse work team. GEP is proud to be an EEO/AA employer M/F/D/V.

GEP's strength is its people. As a leader in procurement and supply chain solutions, we are committed to providing innovative solutions that create a competitive advantage for enterprises worldwide. Our leadership helps us maintain this focus and drives our commitment to deliver the highest value to our clients.

GEP's leadership is a collective of some of the most experienced and accomplished professionals in the industry. Blending together strong business acumen with rich technical expertise, they are leading GEP's global workforce into the future

#WeAreGEP

What makes us really stand out from the crowd?

We value and invest in talent. We respect honesty and individuality. We put our clients and our people first. This is what makes us special, and why GEP is an exceptional place to work.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Engineer - Data Engineering
Senior Data Engineer - Data Engineering

GEP Worldwide • São Paulo

On-site
BRL 643,000 - 965,000
Principal Data Scientist - AI
Principal Data Scientist - AI

GEP • São Paulo

On-site
BRL 150,000 - 230,000
Principal Data Scientist - AI
Principal Data Scientist - AI

GEP Worldwide • São Paulo

On-site
BRL 120,000 - 190,000
Sr. Data Engineer
Sr. Data Engineer

GM Financial • São Paulo

On-site
BRL 180,000 - 280,000
Evergreen - FDE Brazil
Evergreen - FDE Brazil

FDE Times • Brazil

On-site
BRL 320,000 - 520,000
Senior Data Software Engineer
Senior Data Software Engineer

EPAM Systems • Brazil

Remote
BRL 180,000 - 240,000
Healthcare benefits
Paid time off and sick leave
Upskilling, reskilling and certific c a
+2
Lead Data Software Engineer
Lead Data Software Engineer

EPAM Systems • Brazil

Remote
BRL 180,000 - 360,000
Sr. Designated Support Engineer
Sr. Designated Support Engineer

Databricks Inc. • São Paulo

On-site
BRL 180,000 - 240,000
Sr. Designated Support Engineer, Apache Spark
Sr. Designated Support Engineer, Apache Spark

Databricks Inc. • São Paulo

On-site
BRL 180,000 - 300,000
Lead Data Software Engineer (Java+GCP)
Lead Data Software Engineer (Java+GCP)

EPAM Systems Inc • Brazil

On-site
BRL 280,000 - 420,000