Senior Data Engineer

Software International Corporation (M) Sdn Bhd; Kuala Lumpur; On-site; MYR 180,000 - 240,000; Full time

Job summary

Software International Corporation (M) Sdn Bhd in Malaysia is seeking an experienced Data Engineer to architect and deliver scalable data pipelines on Databricks, with strong governance and security via Unity Catalog. You will lead Dalbricks-based data warehouse and lakehouse implementations, optimize PySpark workloads, and drive DevOps practices across CI/CD and GitHub workflows.

You will collaborate with stakeholders across business and technical teams, mentor junior engineers, and participate

Qualifications

• Bachelor's degree in CS/IT/Engineering/Data Science or related field.
• 5-8 years in Data Engineering, Data Warehousing, or Big Data tech.
• Minimum 4+ years hands-on Databricks in enterprise environments.
• Experience designing enterprise Data Warehouse solutions and modern Lakehouse architectures.
• Strong PySpark development and optimization.
• Experience with Unity Catalog, RBAC/ABAC, and data governance.

Responsibilities

• Design, develop, and maintain scalable data pipelines and data products using Databricks and PySpark.
• Build and optimize ETL/ELT solutions for batch and near real-time processing.
• Design and maintain Data Warehouse and Lakehouse solutions for reporting, analytics, and AI/ML use cases.
• Develop enterprise data models with Delta Lake and Delta Tables; implement SCDs.
• Implement data governance with Unity Catalog, RBAC/ABAC, data lineage and auditing.
• Create CI/CD pipelines for Databricks solutions and integrate with GitHub Actions.
• Collaborate with stakeholders across business and technical teams; provide mentorship and production support.

Skills

PySpark

Delta Lake

Unity Catalog

Databricks Workflows

RBAC/ABAC

GitHub & CI/CD

SQL & query optimization

Cloud platforms

Education

Bachelor's Degree in Computer Science/IT/Engineering/Data Science or related field

Tools

Databricks

Delta Lake

Delta Sharing

GitHub Actions

Terraform

Job description

Design, develop, and maintain scalable data pipelines and data products using Databricks and PySpark.

Build and optimize ETL/ELT solutions supporting batch and near real-time data processing.

Design, develop, and maintain enterprise Data Warehouse and Lakehouse solutions that support reporting, analytics, and AI/ML use cases.

Develop and maintain enterprise data models using Delta Lake and Delta Tables.

Design and implement dimensional data models, including Fact and Dimension tables, Star Schema, Snowflake Schema, and Slowly Changing Dimensions (SCD).

Ensure data quality, reliability, scalability, and performance across the data platform.

Implement best practices for code management, testing, deployment, and operational monitoring.

Databricks Platform & Governance

Implement and manage Unity Catalog for centralized governance, data discovery, and security.

Design and maintain governance frameworks utilizing:

Role-Based Access Control (RBAC)

Attribute-Based Access Control (ABAC)

Fine-grained data permissions

Data lineage and auditing

Configure and manage Delta Sharing to support secure external and internal data collaboration.

Support adoption and administration of Databricks Genie, including:

Security and access governance controls

Performance Optimization

Perform advanced PySpark performance tuning and troubleshooting.

Optimize query performance, cluster utilization, partitioning strategies, and workload management.

Identify bottlenecks and proactively improve platform efficiency and cost optimization.

Optimize Data Warehouse and Lakehouse workloads to support high-performance reporting and analytical processing.

DevOps & Automation

Design and implement CI/CD pipelines for Databricks solutions.

Integrate Databricks development lifecycle with GitHub, GitHub Actions, and enterprise DevOps processes.

Automate deployment, testing, code validation, and release management processes.

Establish infrastructure and data engineering best practices.

Stakeholder Management

Engage with business users, data consumers, architects, analysts, and technology leadership to gather requirements and deliver data solutions.

Translate business requirements into scalable technical designs, data models, and platform capabilities.

Communicate effectively with stakeholders across multiple organizational levels.

Work independently while managing priorities and ensuring timely delivery of commitments.

Provide technical guidance and mentorship to junior team members where required.

Production Support

Participate in a rotating production support roster.

Troubleshoot production incidents and prioritize issue resolution within established SLA requirements.

Conduct root cause analysis and implement preventive measures.

Ensure platform stability, reliability, and operational excellence.

Required Qualifications

Experience

Bachelor's Degree in Computer Science, Information Technology, Engineering, Data Science, or a related field.

5-8 years of experience in Data Engineering, Data Warehousing, or Big Data technologies.

Minimum 4+ years of hands-on Databricks experience in enterprise environments.

Experience designing and implementing enterprise Data Warehouse solutions and modern Lakehouse architectures.

Technical Skills

Strong experience in:

PySpark development and optimization

Delta Lake and Delta Tables

Unity Catalog

Databricks Workflows

RBAC and ABAC implementation within Unity Catalog

GitHub and Git-based development workflows

CI/CD implementation using GitHub Actions or equivalent

SQL and advanced query optimization

Cloud platforms (Azure, AWS, or GCP)

Data security, governance, and compliance frameworks

Enterprise Data Warehouse architecture and implementation

Dimensional data modeling (Star Schema and Snowflake Schema)

Fact and Dimension modeling

Slowly Changing Dimensions (SCD Type 1 & Type 2)

Data Warehouse performance tuning and optimization

Additional Technical Knowledge

Data warehouse concepts and methodologies

Lakehouse architecture and Medallion design patterns

Data observability and monitoring

Infrastructure-as-Code (Terraform preferred)

Soft Skills

Strong ownership mindset with high accountability and commitment to delivery.

Ability to work independently with minimal supervision.

Excellent analytical and problem-solving skills.

Strong communication and stakeholder management capabilities.

Ability to work effectively with stakeholders across business and technical functions.

Ability to manage multiple priorities in a fast-paced environment.

Collaborative team player with a proactive and customer-focused attitude.

Willingness to participate in production support and on-call rotation schedules.

Preferred Qualifications

Databricks Certified Data Engineer Associate or Professional certification.

Experience implementing enterprise data governance frameworks.

Experience supporting large-scale Lakehouse and Data Warehouse architectures.

Experience working with regulated industries and compliance requirements.

Knowledge of Data Mesh and modern data platform architectures.

Experience integrating Databricks with Power BI, Tableau, or other BI and analytics platforms.

Success Factors

The successful candidate will:

Be the go-to Databricks engineering expert within the team.

Drive governance and security best practices through Unity Catalog.

Deliver reliable, scalable, and high-performing data solutions and enterprise data warehouse platforms.

Partner effectively with business and technical stakeholders.

Demonstrate strong ownership, accountability, and operational excellence.

Contribute to continuous improvement of the organization's modern data platform capabilities.

Unlock job insights

Hirer responsiveness Salary match Number of applicants

Computer Software & Networking 101-1,000 employees

Software International Corporation (M) Sdn Bhd (SI), established in 1995, was one of Malaysia’s first MSC-status IT service providers. SI delivers Infrastructure Management and Application System Solutions across industries such as Insurance, Telco, Construction, Data Centers, and Government. A joint venture with a Global Fortune 100 Financial Services Company in 1998 strengthened its global capabilities. Today, SI serves diversified worldwide clients, including long-term Fortune 100 partners.

Software International Corporation (M) Sdn Bhd (SI), established in 1995, was one of Malaysia’s first MSC-status IT service providers. SI delivers Infrastructure Management and Application System Solutions across industries such as Insurance, Telco, Construction, Data Centers, and Government. A joint venture with a Global Fortune 100 Financial Services Company in 1998 strengthened its global capabilities. Today, SI serves diversified worldwide clients, including long-term Fortune 100 partners.