Databricks Engineer

Espire Infolabs (Singapore) Pte Ltd; Região Norte; On-site; BRL 467,000 - 675,000; Full time

Job summary

Espire Infolabs (Singapore) Pte Ltd is seeking a skilled Databricks Engineer to design, develop, and maintain scalable data solutions on the Databricks Lakehouse Platform. The ideal candidate will have hands-on experience with Databricks, Spark, Python, SQL, and Delta Lake to build reliable data pipelines.

The role involves working with Azure, AWS, or GCP, implementing data quality, governance, and security, and collaborating with architects, scientists, and BI teams to deliver high-performance

Qualifications

• 3–5 years of experience in Data Engineering.
• Strong hands-on experience with Databricks.
• Strong knowledge of Apache Spark / PySpark.
• Proficient in Python and SQL.
• Experience with Delta Lake and ETL/ELT concepts.
• Experience building and managing data pipelines.
• Knowledge of data warehousing and data modelling.
• Experience with cloud platforms, preferably Microsoft Azure.
• Experience with cloud storage (ADLS, S3, GCS).
• Familiarity with Git and CI/CD practices.
• Strong troubleshooting and analytical skills.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.
• Develop ETL/ELT pipelines using PySpark, Python, and SQL.
• Build and maintain Delta Lake tables and data workflows.
• Work with Databricks Lakehouse architecture and components.
• Develop batch and near-real-time data processing solutions.
• Ingest and transform data from databases, APIs, files, and other sources.
• Implement data cleansing, transformation, validation, and quality checks.
• Optimize Spark jobs and Databricks workloads for performance and cost.
• Work with cloud storage and data services across Azure, AWS, or GCP.
• Implement data security, access controls, and governance.
• Collaborate with Data Architects, Data Scientists, BI Developers, and stakeholders.
• Troubleshoot pipeline failures and data-quality issues.
• Develop and maintain technical documentation.
• Participate in code reviews, testing, deployment, and production support.
• Follow Agile practices and contribute to continuous improvement.

Skills

Databricks

Apache Spark

PySpark

Python

SQL

Delta Lake

ETL/ELT

Data pipelines

Data warehousing

CI/CD

Troubleshooting

Tools

Azure

Azure Data Lake Storage

Amazon S3

Google Cloud Storage

Git

CI/CD

Databricks Workflows

Unity Catalog

Terraform

Job description

About the Role

We are looking for a skilled Databricks Engineer todesign, develop, and maintain scalable data engineering solutions using the DatabricksLakehouse Platform.

The ideal candidate will have strong hands‑on experiencewith Databricks, Apache Spark, Python, SQL, Delta Lake, and cloud dataplatforms, with the ability to build reliable and high‑performance datapipelines.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.
• Develop ETL/ELT pipelines using PySpark, Python, and SQL.
• Build and maintain Delta Lake tables and data processing workflows.
• Work with Databricks Lakehouse architecture and related data engineering components.
• Develop batch and, where required, near-real-time data processing solutions.
• Ingest and transform data from databases, APIs, files, and other data sources.
• Implement data cleansing, transformation, validation, and quality checks.
• Optimise Spark jobs and Databricks workloads for performance and cost efficiency.
• Work with cloud storage and data services across Azure, AWS, or GCP.
• Implement data security, access controls, and governance within the data platform.
• Collaborate with Data Architects, Data Scientists, BI Developers, and business stakeholders.
• Troubleshoot data pipeline failures and resolve performance and data-quality issues.
• Develop and maintain technical documentation for data pipelines and solutions.
• Participate in code reviews, testing, deployment, and production support.
• Follow Agile development practices and contribute to continuous improvement.

Required Skills & Experience

• 3–5 years of experience in Data Engineering.
• Strong hands‑on experience with Databricks.
• Strong knowledge of:
• Apache Spark / PySpark
• Python
• SQL
• Delta Lake
• ETL/ELT concepts
• Experience developing and managing data pipelines.
• Good understanding of data warehousing and data modelling concepts.
• Experience working with cloud platforms, preferably Microsoft Azure.
• Experience with cloud storage such as Azure Data Lake Storage (ADLS), Amazon S3, or Google Cloud Storage.
• Experience with relational and/or NoSQL databases.
• Good understanding of data quality, validation, and governance.
• Familiarity with Git and CI/CD practices.
• Strong troubleshooting and analytical skills.

Good to Have

• Databricks Certified Data Engineer Associate/Professional certification.
• Experience with Azure Data Factory.
• Experience with Microsoft Fabric.
• Knowledge of Unity Catalog and Databricks governance.
• Experience with Databricks Workflows and job orchestration.
• Experience with streaming technologies such as Kafka or Structured Streaming.
• Experience with Power BI or other BI platforms.
• Exposure to Machine Learning workflows on Databricks.
• Experience with Terraform or Infrastructure as Code.
• Knowledge of DevOps and CI/CD pipelines.

Candidate Profile

The ideal candidate should be a hands‑on Databricks/Data Engineer capable of independently developing data pipelines, troubleshooting production issues, optimising Spark workloads, and collaborating with technical and business teams.