Databricks Platform Engineer — Delta Lake & ETL Pipelines

Accenture in the Philippines; Philippines; On-site; PHP 700,000 - 1,100,000; Full time

Job summary

Accenture in the Philippines is seeking a data engineer to design, develop, and maintain data solutions for data generation, collection, and processing. The role involves creating data pipelines, ensuring data quality, and implementing ETL/ELT processes to migrate and deploy data across systems.

The candidate will have hands-on experience with PySpark, Python, and Databricks, and must be proficient in at least one cloud platform (AWS, GCP, or Azure).

Qualifications

• Minimum 1 year of experience is required.
• Develop high-quality, scalable ETL/ELT pipelines using Databricks technologies including Delta Lake, Auto Loader, and DLT.
• Excellent programming and debugging skills in Python.
• Strong hands-on experience with PySpark to build efficient data transformation and validation logic.
• Must be proficient in at least one cloud platform: AWS, GCP, or Azure.
• Create modular dbx functions for transformation, PII masking, and validation logic — reusable across DLT and notebook pipelines.
• Implement ingestion patterns using Auto Loader with checkpointing and schema evolution for structured and semi-structured data.
• Build secure and observable DLT pipelines with DLT Expectations, supporting Bronze/Silver/Gold medallion layering.
• Configure Unity Catalog: set up catalogs, schemas, user/group access, enable audit logging, and define masking for PII fields.
• Enable secure data access across domains and workspaces via Unity Catalog External Locations, Volumes, and lineage tracking.
• Access and utilize data assets from the Databricks Marketplace to support enrichment, model training, or benchmarking.
• Collaborate with data sharing stakeholders to implement Delta Sharing - both internally and externally.
• Integrate Power BI/Tableau/Looker with Databricks using optimized connectors (ODBC/JDBC) and Unity Catalog security controls.
• Build stakeholder-facing SQL Dashboards within Databricks to monitor KPIs, data pipeline health, and operational SLAs.
• Prepare GenAI-compatible datasets: manage vector embeddings, index with Databricks Vector Search, and use Feature Store with MLflow.
• Package and deploy pipelines using Databricks Asset Bundles through CI/CD pipelines in GitHub or GitLab.
• Troubleshoot, tune, and optimize jobs using Photon engine and serverless compute, ensuring cost efficiency and SLA reliability.
• Experience with cloud-based services relevant to data engineering, data storage, data processing, data warehousing, real-time streaming, and serverless computing.
• Hands on Experience in applying Performance optimization technique
• Understanding data modeling and data warehousing principles is essential

Responsibilities

• Design, develop and maintain data solutions for data generation, collection, and processing.
• Create data pipelines, ensure data quality, and implement ETL/ELT processes to migrate and deploy data across systems.

Tools

Power BI

Tableau

Looker

Airflow

CI/CD

Job description

Accenture in the Philippines is seeking a data engineer to design, develop, and maintain data solutions for data generation, collection, and processing. The role involves creating data pipelines, ensuring data quality, and implementing ETL/ELT processes to migrate and deploy data across systems.

The candidate will have hands-on experience with PySpark, Python, and Databricks, and must be proficient in at least one cloud platform (AWS, GCP, or Azure).