Databricks Data Engineer

Accenture; Metro Manila; On-site; PHP 600,000 - 1,200,000; Full time

Job summary

Accenture is seeking an experienced Databricks Data Engineer to design, build, and optimize enterprise data solutions that support analytics, reporting, ML, and Generative AI use cases.

You will develop scalable ETL and ELT pipelines using Databricks, Python, and PySpark, working with Delta Lake, Auto Loader, and Delta Live Tables. Collaborate with teams to enable secure data sharing and BI integrations.

Qualifications

• Must have hands-on experience with Databricks data engineering workflows and cloud-based data platforms.
• Proficient in Python and PySpark for building scalable ETL/ELT pipelines.
• Strong understanding of data governance, security, and CI/CD practices.

Responsibilities

• Design, develop, and maintain scalable data solutions for analytics, reporting, ML, and Generative AI use cases.
• Build ETL/ELT pipelines using Databricks, Delta Lake, Auto Loader, and Delta Live Tables.
• Write, test, and maintain high-quality Python code; implement data transformation and validation with PySpark.
• Create modular Databricks functions for data transformation and PII masking.
• Develop reusable components for notebooks and Delta Live Tables pipelines.
• Configure Unity Catalog and secure data sharing via Delta Sharing; ensure access control and logging.
• Connect Databricks with BI tools (Power BI, Tableau, Looker) via connectors and Unity Catalog security controls.

Skills

Databricks

Python

PySpark

CI/CD

Data modeling

Tools

Delta Lake

Auto Loader

Delta Live Tables

Unity Catalog

Databricks Asset Bundles

Job description

Build scalable cloud data solutions with Databricks

Join Accenture as a Databricks Data Engineer and help design, build, and optimize enterprise data solutions that support analytics, reporting, machine learning, and Generative AI use cases.

You will develop scalable ETL and ELT pipelines using Databricks, Python, and PySpark. You will also work with Delta Lake, Auto Loader, Delta Live Tables, Unity Catalog, cloud technologies, business intelligence tools, and CI/CD practices.

This role is ideal for an experienced Data Engineer who enjoys solving complex data challenges, improving pipeline performance, and building secure and reliable cloud data platforms.

What you’ll do

• Design, develop, and maintain solutions for data generation, collection, processing, migration, and deployment.

• Build scalable ETL and ELT pipelines using Databricks, Delta Lake, Auto Loader, and Delta Live Tables.

• Write, test, debug, and maintain high-quality Python code.

• Use PySpark to build efficient data transformation and validation processes.

• Create modular Databricks functions for data transformation, personally identifiable information masking, and validation.

• Develop reusable components for Databricks notebooks and Delta Live Tables pipelines.

• Implement ingestion patterns using Auto Loader, checkpointing, and schema evolution.

• Process structured and semi-structured data from multiple sources.

• Build secure and observable Delta Live Tables pipelines using DLT Expectations.

• Implement Bronze, Silver, and Gold data layers following medallion architecture principles.

• Configure Unity Catalog, including catalogs, schemas, user and group access, audit logging, and masking for personally identifiable information.

• Enable secure access across domains and Databricks workspaces using External Locations, Volumes, and lineage tracking.

• Use data assets from Databricks Marketplace for enrichment, model training, or benchmarking.

• Collaborate with stakeholders to implement secure internal and external data sharing through Delta Sharing.

• Connect Databricks with Power BI, Tableau, or Looker using ODBC or JDBC connectors and Unity Catalog security controls.

• Build Databricks SQL dashboards for business KPIs, pipeline health, and operational service-level commitments.

• Prepare datasets for Generative AI use cases using vector embeddings, Databricks Vector Search, Feature Store, and MLflow.

• Package and deploy data pipelines using Databricks Asset Bundles and CI/CD workflows in GitHub or GitLab.

• Troubleshoot and optimize data workloads using Photon and serverless compute.

• Apply performance optimization techniques to improve processing speed, reliability, cost efficiency, and service-level performance.

• Work with cloud services for data storage, data processing, data warehousing, real-time streaming, and serverless computing.