Lead Data Software Engineer

Jobgether; Brasil; On-site; BRL 350,000 - 600,000; Full time

Benefits offered by this job

Healthcare benefits

Employee financial programs

Paid time off and sick leave

Upskilling and professional certs

LinkedIn Learning access

Global career development

Job summary

Jobgether is seeking a Lead Data Software Engineer based in Brazil to design and maintain scalable data pipelines and governance-enabled data platforms. You will lead a team, collaborate across departments, and apply modern tools like PySpark, Spark SQL, and Databricks to deliver reliable data solutions in an Agile environment.

You will drive ingestion from diverse sources, ensure data quality, and participate in code reviews while advancing engineering standards and documentation.

Qualifications

• At least 5 years of relevant professional experience.
• At least 1 year of experience leading and managing development teams.
• Strong experience with PySpark and Spark SQL for large-scale data processing.
• Hands-on experience with Databricks Lakeflow pipelines and Jobs.
• Proficiency with Delta Lake and Unity Catalog for data governance.
• Experience implementing data quality controls and source reconciliation.
• Unit testing with pytest to validate code quality.
• Git and Databricks Asset Bundles for version control and deployment.

Responsibilities

• Design, build, and maintain scalable data pipelines across multiple systems.
• Develop and manage ingestion pipelines from diverse source systems.
• Write clean, maintainable, thoroughly tested code.
• Collaborate with cross-functional teams to translate requirements into solutions.
• Monitor pipeline performance and maintain system reliability.
• Participate in code reviews and uphold engineering standards.
• Investigate and resolve data quality issues.
• Contribute to data architecture and development practices.
• Create and maintain technical documentation.
• Participate in Agile ceremonies and team planning.
• Provide technical leadership to the development team.

Skills

Data engineering

Team leadership

Agile

Code quality

Tools

PySpark

Spark SQL

Databricks

Delta Lake

Unity Catalog

Auto Loader

JDBC/CDC

Git

Databricks Asset Bundles

pytest

Job description

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Lead Data Software Engineer based in Brazil.

We are looking for an experienced Lead Data Software Engineer to design and build resilient, scalable data systems that support critical business decisions. You will develop and maintain robust data pipelines, manage ingestion from diverse source systems, and ensure data accuracy across complex environments. Working with modern technologies such as PySpark, Spark SQL, and Databricks, you will help deliver reliable and well-governed data solutions at scale. You will also contribute to engineering standards, code quality, and continuous improvements across the data platform. In this role, you will combine hands‑on technical expertise with team leadership and close collaboration across departments. It is an opportunity to solve challenging data problems while making a lasting impact in a quality‑focused, Agile environment.

Accountabilities

• Design, build, and maintain scalable data pipelines that ensure reliable data flow across multiple systems.

• Develop and manage ingestion pipelines that collect and process data from diverse source systems.

• Write clean, maintainable, high-quality, and thoroughly tested code based on strong software engineering principles.

• Collaborate with cross‑functional teams to understand business and data requirements and translate them into effective technical solutions.

• Monitor pipeline performance, identify potential issues proactively, and maintain system reliability.

• Participate in code reviews, uphold engineering quality standards, and share technical knowledge with team members.

• Investigate and resolve data quality issues to ensure the accuracy, consistency, and reliability of datasets.

• Contribute to the continuous improvement of data architecture, development practices, and engineering standards.

• Create and maintain technical documentation to support knowledge sharing, operational continuity, and onboarding.

• Participate in Agile ceremonies, including sprint planning and regular team meetings.

• Provide technical leadership and support the development team in delivering reliable data engineering solutions.

Requirements:

• At least 5 years of relevant professional experience.

• At least 1 year of experience leading and managing development teams.

• Strong experience with PySpark and Spark SQL for building and managing large‑scale data processing workflows.

• Hands‑on experience developing and managing Databricks Lakeflow pipelines and Jobs for data orchestration.

• Proficiency with Auto Loader and JDBC/CDC ingestion strategies, including experience integrating data from DB2 and SQL Server.

• Strong knowledge of Delta Lake and Unity Catalog for data storage, management, and governance.

• Experience implementing data quality controls and source reconciliation processes to ensure trustworthy data.

• Background writing unit tests with pytest to validate code quality and reliability.

• Working knowledge of Git and Databricks Asset Bundles for version control and deployment management.

• Upper‑intermediate English proficiency or higher (B2).

• Experience participating in DataStage migration projects.

• Experience extracting and integrating data from Dynamics 365 or Dataverse.

• Databricks certification.

Benefits:

• Opportunity to work on international projects for leading global brands.

• Collaboration with highly skilled and diverse teams across different locations.

• Healthcare benefits.

• Employee financial programs.

• Paid time off and sick leave.

• Upskilling, reskilling, and professional certification opportunities.

• Unlimited access to LinkedIn Learning, including more than 22,000 courses.

• Global career development opportunities.

• Volunteer and community involvement initiatives.

• Employee Groups and opportunities to connect with colleagues.

• An award‑winning workplace culture recognized by Glassdoor, Newsweek, and LinkedIn.