Databricks Unified Data Analytics Platform Engineer

Accenture Inc.; Metro Manila; On-site; PHP 700,000 - 1,200,000; Full time

Job summary

Accenture Inc. is seeking a Data Engineer to design, develop and maintain data solutions for data generation, collection, and processing. You will build scalable ETL/ELT pipelines using Databricks, Delta Lake, and PySpark, ensuring data quality and enabling BI and analytics across cloud platforms.

Responsibilities include implementing Delta Sharing, configuring Unity Catalog, and deploying pipelines via CI/CD. Collaboration with stakeholders to monitor KPIs and SLA compliance is essential.

Qualifications

• Advanced knowledge of ETL/ELT design and data pipelines.
• Proficiency in Python programming for data processing and validation.
• Hands-on PySpark experience for large-scale data transformations.
• Experience with cloud platforms: AWS, GCP, or Azure.
• Experience with Databricks Delta Lake and Delta Sharing.
• Ability to build modular data transformation functions and automate pipelines.

Responsibilities

• Develop scalable ETL/ELT pipelines using Databricks technologies (Delta Lake, Auto Loader, DLT).
• Implement data quality checks and validation logic across pipelines.
• Configure Unity Catalog, access controls, and auditing for data safety.
• Create SQL dashboards and BI connectors (ODBC/JDBC) to monitor KPIs.
• Collaborate with stakeholders to enable Delta Sharing internally and externally.
• Package pipelines for CI/CD deployment using GitHub or GitLab.

Skills

Databricks

Python

PySpark

Delta Lake

Auto Loader

DLT

Unity Catalog

Cloud Platforms

Data Warehousing

CI/CD

Tools

Databricks

Power BI

Tableau

Looker

Unity Catalog

ODBC/JDBC connectors

GitHub/GitLab

Job description

Design, develop and maintain data solutions for data generation, collection, and processing. Create data pipelines, ensure data quality, and implement ETL (extract, transform and load) processes to migrate and deploy data across systems.

Priority Skill

Data Warehousing (DW)

Responsibilities:

• Develop high-quality, scalable ETL/ELT pipelines using Databricks technologies including Delta Lake, Auto Loader, and DLT.
• Excellent programming and debugging skills in Python.
• Strong hands on experience with PySpark to build efficient data transformation and validation logic.
• Must be proficient in at least one cloud platform: AWS, GCP, or Azure.
• Create modular dbx functions for transformation, PII masking, and validation logic — reusable across DLT and notebook pipelines.
• Implement ingestion patterns using Auto Loader with checkpointing and schema evolution for structured and semi-structured data.
• Build secure and observable DLT pipelines with DLT Expectations, supporting Bronze/Silver/Gold medallion layering.
• Configure Unity Catalog: set up catalogs, schemas, user/group access, enable audit logging, and define masking for PII fields.
• Enable secure data access across domains and workspaces via Unity Catalog External Locations, Volumes, and lineage tracking.
• Access and utilize data assets from the Databricks Marketplace to support enrichment, model training, or benchmarking.
• Collaborate with data sharing stakeholders to implement Delta Sharing - both internally and externally.
• Integrate Power BI/Tableau/Looker with Databricks using optimized connectors (ODBC/JDBC) and Unity Catalog security controls.
• Build stakeholder-facing SQL Dashboards within Databricks to monitor KPIs, data pipeline health, and operational SLAs.
• Prepare GenAI-compatible datasets: manage vector embeddings, index with Databricks Vector Search, and use Feature Store with MLflow.
• Package and deploy pipelines using Databricks Asset Bundles through CI/CD pipelines in GitHub or GitLab.
• Troubleshoot, tune, and optimize jobs using Photon engine and serverless compute, ensuring cost efficiency and SLA reliability.
• Experience with cloud-based services relevant to data engineering, data storage, data processing, data warehousing, real-time streaming, and serverless computing.
• Hands on Experience in applying Performance optimization techniques
• Understanding data modeling and data warehousing principles is essential.

Good to Have:

1. Certifications: Databricks Certified Professional or similar certifications.
2. Machine Learning: Knowledge of machine learning concepts and experience with popular ML libraries.
3. Knowledge of big data processing (e.g., Spark, Hadoop, Hive,Kafka)
4. Data Orchestration: Apache Airflow.
5. Knowledge of CI/CD pipelines and DevOps practices in a cloud environment.
6. Experience with ETL tools like Informatica, Talend, Matillion, or Fivetran.
7. Familiarity with dbt (Data Build Tool)

Minimum 3 year(s) of experience is required

NOTE: Open demands are subject to change.