Senior Data Engineer - Data Engineering

GEP Worldwide; São Paulo; Presencial; BRL 643 000 - 965 000; Tempo integral

Resumo da oferta

GEP Worldwide is seeking a Senior Data Engineer to shape our Databricks lakehouse architecture and drive strategic data initiatives. You will lead the design and implementation of complex data solutions, set engineering standards, and mentor engineers across cross-functional teams.

The role emphasizes performance, governance, and scalable pipelines. You will collaborate with analysts, product managers, and stakeholders to translate business needs into robust data solutions, ensuring data quality

Qualificações

• Expert-level hands-on experience with Databricks, Lakeflow, and Delta Lake architectures.
• Deep understanding of Spark and distributed data processing including performance tuning.
• Expert-level Python and SQL skills for data engineering (PySpark, Pandas).
• Strong experience with Unity Catalog governance, access control, and data lineage.
• Experience with CI/CD for data solutions and automated testing (pytest).
• Knowledge of data modeling, lakehouse/medallion architecture and Delta Lake optimization.

Responsabilidades

• Lead design, development, and optimization of scalable data solutions on Databricks using Python, SQL, and PySpark.
• Architect lakehouse solutions with Delta Lake and medallion architecture.
• Define and govern data assets in Unity Catalog, including schemas and access control.
• Design and build batch and incremental pipelines with Lakeflow and MERGE/CDC patterns.
• Own CI/CD for Databricks solutions with DABs, Git, and automated testing.
• Mentor junior engineers and participate in technical interviews and reviews.

Conhecimentos

Databricks Lakehouse

Python

SQL

PySpark

Delta Lake

Unity Catalog

Data CI/CD

Azure Databricks

Cloud platforms

Mentorship

Ferramentas

Lakeflow

Git

Databricks Asset Bundles

Selenium

Descrição da oferta de emprego

Who We Are

GEP, with over 30 offices internationally, is the fastest growing procurement and supply chain solutions firm – consulting, software and managed services. GEP has succeeded by being smart and creative; solving complex problems and finding opportunities for the world’s largest organizations.

Welcome to GEP

How You Grow At GEP

We recognize people who make a genuine difference, work hard and drive achievements. Results are noticed and rewarded. It’s how you will grow a career at GEP, and in a much shorter time frame than at other firms.

Celebrating Everyone

GEP succeeds through the ideas and creativity of our team members so we embrace people of all experiences, nationalities, abilities, cultures, races, gender identities, sexual orientations and ages. What makes you unique and different is celebrated and will help GEP stand out even more. And we are a women-founded and -owned company so our foundation is making GEP a great place to work for women, a place where women can learn, advance and give back.

What You Will Do

As a Senior Data Engineer at GEP, you will play a pivotal role in shaping our Databricks lakehouse architecture and driving strategic data initiatives that support our business intelligence and analytics. You will lead the design and implementation of complex data solutions, set engineering standards, mentor other engineers, and collaborate with cross-functional teams to ensure our data platform meets the highest standards of performance, reliability, and governance. You will:

• Lead the design, development, and optimization of scalable data solutions on Databricks using Python, SQL, and PySpark.
• Architect lakehouse solutions based on Delta Lake and medallion architecture, defining data models, layer boundaries, and performance strategies.
• Define and govern data assets in Unity Catalog, including catalog and schema structure, access permissions, and lineage.
• Design and build batch and incremental pipelines with Lakeflow Jobs and Lakeflow Declarative Pipelines, including MERGE/CDC patterns and data-quality expectations.
• Own CI/CD for our Databricks solutions, packaging and deploying jobs and pipelines across environments with Databricks Asset Bundles (DABs), Git, and automated testing.
• Establish data-quality, observability, and data-contract practices that ensure the accuracy, consistency, and agreed service levels of business-critical datasets.
• Optimize Databricks workloads for performance and cost, including compute selection (classic and serverless), Spark tuning, and Delta table maintenance.
• Provide technical leadership in architectural decisions, code reviews, and the troubleshooting of critical pipeline and platform issues.
• Mentor junior and mid-level engineers, fostering a culture of learning and technical excellence, and take part in technical interviews and assessments.
• Collaborate with analysts, product managers, and business stakeholders to translate business needs into data solutions and communicate technical trade-offs.

What Should You Bring

• Expert-level hands-on experience with Databricks, including Lakeflow Jobs, Lakeflow Declarative Pipelines, Spark DataFrames, and Delta Lake.
• Deep understanding of Spark and distributed data processing, including partitioning, joins, shuffles, and performance tuning.
• Expert-level Python and SQL skills for data engineering, including PySpark, Pandas, and writing testable, production-grade code.
• Strong experience with Unity Catalog governance, including catalog design, access control, and lineage.
• Strong experience with CI/CD for data solutions, including Databricks Asset Bundles (DABs), Git-based workflows, and automated testing (e.g., pytest).
• Solid knowledge of data modeling, lakehouse and medallion architecture, incremental processing (MERGE, CDC), and Delta Lake optimization.
• Knowledge designing data-quality frameworks, observability, and data contracts for production pipelines.
• Expertise with Azure Databricks is strongly preferred. Experience with AWS, Google Cloud, or another cloud platform is also valuable.
• Familiarity with AI-assisted development tools and Databricks AI Functions is a plus.
• Kkoeledge with web scraping tools and techniques (e.g., BeautifulSoup, Scrapy, Selenium) is a plus.
• Knowledge of procurement, public procurement, or foreign trade (import/export) data is a plus.
• Proven ability to lead projects in agile environments, balancing technical and business priorities.
• Excellent communication and mentorship skills to guide engineers and present solutions to stakeholders.

Are You One of Us

GEP is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, ethnicity, color, national origin, religion, sex, protected veteran status, disability status, or any other characteristics protected by federal, state or local law. We are committed to hiring and valuing a global diverse work team. GEP is proud to be an EEO/AA employer M/F/D/V.