Databricks Engineer

Espire Infolabs; Manaus; On-site; BRL 120,000 - 180,000; Full time

Job summary

Espire Infolabs in Manaus is seeking a hands-on Databricks/Data Engineer to design, develop, and maintain scalable data pipelines on the Databricks Lakehouse Platform. The role emphasizes building reliable ETL/ELT processes and optimizing Spark workloads.

The ideal candidate will have 3–5 years of experience with Databricks, Spark, Python, SQL, Delta Lake, and cloud storage across Azure, AWS, or GCP, plus a solid grasp of data governance and CI/CD practices.

Qualifications

• 3–5 years of experience in data engineering.
• Strong hands-on experience with Databricks.
• Strong knowledge of Apache Spark, PySpark, Python, SQL, Delta Lake, and ETL/ELT concepts.
• Experience developing and managing data pipelines.
• Good understanding of data warehousing and data modelling.
• Experience with cloud platforms, preferably Microsoft Azure.
• Experience with cloud storage such as ADLS, S3, or GCS.
• Experience with relational and/or NoSQL databases.
• Good understanding of data quality, validation, and governance.
• Familiarity with Git and CI/CD practices.
• Databricks certification is a plus.
• Experience with Azure Data Factory, Microsoft Fabric, Unity Catalog, and Databricks governance.
• Experience with Databricks Workflows and job orchestration.
• Experience with streaming technologies such as Kafka or Structured Streaming.
• Experience with Power BI or other BI platforms.
• Exposure to ML workflows on Databricks.
• Experience with Terraform or Infrastructure as Code.
• Knowledge of DevOps and CI/CD pipelines.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.
• Develop ETL/ELT pipelines using PySpark, Python, and SQL.
• Build and maintain Delta Lake tables and data processing workflows.
• Work with Databricks Lakehouse architecture and related data engineering components.
• Develop batch and near-real-time data processing solutions.
• Ingest and transform data from databases, APIs, files, and other data sources.
• Implement data cleansing, transformation, validation, and quality checks.
• Optimize Spark jobs and Databricks workloads for performance and cost efficiency.
• Work with cloud storage across Azure, AWS, or GCP; implement data security and governance.
• Collaborate with Data Architects, Data Scientists, BI Developers, and business stakeholders.
• Troubleshoot data pipeline failures and resolve data-quality issues.
• Develop and maintain technical documentation for data pipelines and solutions.
• Participate in code reviews, testing, deployment, and production support.
• Follow Agile development practices and contribute to continuous improvement.

Skills

Databricks

Apache Spark

Python

SQL

Delta Lake

ETL/ELT concepts

Data pipelines

Data warehousing

Azure/cloud platforms

Git/CI-CD

Kafka (streaming)

Power BI / BI platforms

Terraform / IaC

Databricks governance

Databricks Workflows

Tools

Azure Data Lake Storage (ADLS)

Amazon S3

Google Cloud Storage

Relational databases

NoSQL databases

Job description

About the Role

We are looking for a skilledDatabricks Engineertodesign, develop, and maintain scalable data engineering solutions using theDatabricks Lakehouse Platform.

The ideal candidate will have strong hands‐on experiencewithDatabricks, Apache Spark, Python, SQL, Delta Lake, and cloud dataplatforms , with the ability to build reliable and high‐performance datapipelines.

Key Responsibilities

Design, develop, and maintain scalable data pipelines usingDatabricks and Apache Spark.

Develop ETL/ELT pipelines usingPy Spark, Python, and SQL.Build and maintainDelta Laketables and data processing workflows.

Work with DatabricksLakehouse architectureand related data engineering components.

Develop batch and, where required, near-real-time data processing solutions.

Ingest and transform data from databases, APIs, files, and other data sources.

Implement data cleansing, transformation, validation, and quality checks.

Optimise Spark jobs and Databricks workloads for performance and cost efficiency.

Work with cloud storage and data services acrossAzure, AWS, or GCP.Implement data security, access controls, and governance within the data llaborate with Data Architects, Data Scientists, BI Developers, and business stakeholders.

Troubleshoot data pipeline failures and resolve performance and data‑quality issues.

Develop and maintain technical documentation for data pipelines and solutions.

Participate in code reviews, testing, deployment, and production support.

Follow Agile development practices and contribute to continuous improvement.

Required

Skills & Experience

3–5 years of experience inData Engineering.

Strong hands‐on experience withDatabricks.

Strong knowledge of: Apache Spark / Py SparkPythonSQLDelta LakeETL/ELT conceptsExperience developing and managing data pipelines.

Good understanding of data warehousing and data modelling concepts.

Experience working with cloud platforms, preferablyMicrosoft Azure.

Experience with cloud storage such asAzure Data Lake Storage (ADLS), Amazon S3, or Google Cloud Storage.

Experience with relational and/or No SQL databases.

Good understanding of data quality, validation, and governance.

Familiarity with Git and CI/CD practices.

Strong troubleshooting and analytical skills. Good to HaveDatabricks Certified Data Engineer Associate/Professionalcertification.

Experience withAzure Data Factory.

Experience withMicrosoft Fabric.

Knowledge ofUnity Catalogand Databricks governance.

Experience with Databricks Workflows and job orchestration.

Experience with streaming technologies such asKafkaor Structured Streaming.

Experience with Power BI or other BI platforms.

Exposure to Machine Learning workflows on Databricks.

Experience with Terraform or Infrastructure as Code.

Knowledge of Dev Ops and CI/CD pipelines.

Candidate Profile

The ideal candidate should be a hands‐onDatabricks/Data Engineercapable of independently developing data pipelines, troubleshooting production issues, optimising Spark workloads, and collaborating with technical and business teams.