Senior Data Engineer

Software International Corporation; Kuala Lumpur; Hybrid; MYR 180,000 - 320,000; Full time

Benefits offered by this job

Up to 24 Days Annual Leave

Flexible & Hybrid Working

Performance Bonus

Enhanced EPF Contributions

Training & Development (In-House & Ex-

Travel Allowance & Expense Benefits

Company Trips & Team Events

Long Service Rewards

Job summary

Software International Corporation seeks a senior data engineer to design, develop, and optimize scalable data pipelines and data products using Databricks and PySpark in Kuala Lumpur. The role covers building enterprise Data Warehouses and Lakehouses, governance, and performance tuning for high-volume analytics.

Collaborate with business users and data stewards to deliver robust data platforms, automate deployments, and ensure data quality, security, and scalability across the organization.

Qualifications

• Bachelor's degree in Computer Science, Information Technology, Engineering, Data Science, or a related field.
• 8+ years of experience in Data Engineering, Data Warehousing, or Big Data technologies.
• Minimum 4+ years of hands-on Databricks experience in enterprise environments.
• Experience designing and implementing enterprise Data Warehouse solutions and modern Lakehouse architectures.

Responsibilities

• Design, develop, and maintain scalable data pipelines and data products using Databricks and PySpark.
• Build and optimize ETL/ELT solutions supporting batch and near real-time data processing.
• Design, develop, and maintain enterprise Data Warehouse and Lakehouse solutions for reporting, analytics, and AI/ML use cases.
• Develop and maintain enterprise data models using Data Lake and Delta Tables.
• Implement dimensional data models (Star/Snowflake, SCD) across platforms.
• Ensure data quality, security, and performance across the data platform.
• Implement CI/CD pipelines and DevOps practices for Databricks solutions.
• Engage with stakeholders to translate requirements into scalable technical designs.

Skills

Data engineering

Big data technologies

ETL/ELT design

Stakeholder communication

Education

Bachelor's degree in Computer Science, IT, Engineering, Data Science, or related field

Tools

Databricks

SQL

Python

Delta Lake

Job description

• Design, develop, and maintain scalable data pipelines and data products using Databricks and PySpark.
• Build and optimize ETL/ELT solutions supporting batch and near real-time data processing.
• Design, develop, and maintain enterprise Data Warehouse and Lakehouse solutions that support reporting, analytics, and AI/ML use cases.
• Develop and maintain enterprise data models using Data Lake and Delta Tables.
• Design and implement dimensional data models, including Fact and Dimension tables, Star Schema, Snowflake Schema, and Slowly Changing Dimensions (SCD).
• Ensure data quality, scalability, security, and performance across the data platform.
• Implement best practices for code management, testing, deployment, and operational monitoring.

Databricks Platform & Governance

• Implement and manage Unity Catalog for centralized governance, data discovery, and security.
• Design and enforce governance frameworks including:
• Role-Based Access Control (RBAC)
• Attribute-Based Access Control (ABAC)
• Fine-grained data permissions
• Data lineage and auditing
• Configure and manage Data Sharing to support secure external and internal data collaboration.
• Support the adoption and utilization of Databricks Genie, including:
• Security and access governance controls

Performance Optimization

• Perform advanced PySpark performance tuning and troubleshooting.
• Optimize query performance, cluster utilization, partitioning strategies, and workload management.
• Identify bottlenecks and proactively improve platform efficiency and cost optimization.
• Optimize Data Warehouse and Lakehouse workloads to support high-performance reporting and analytics processing.

DevOps & Automation

• Design and implement CI/CD pipelines for Databricks solutions.
• Integrate Databricks development lifecycle with GitHub, GitHub Actions, and enterprise DevOps processes.
• Automate deployment, testing, code validation, and release management processes.
• Establish infrastructure and data engineering best practices.

Stakeholder Management

• Engage with business users, data consumers, architects, analysts, and technology leadership to gather requirements and deliver data solutions.
• Translate business requirements into scalable technical designs, data models, and platform capabilities.
• Communicate effectively with stakeholders across multiple organizational levels.
• Work independently while managing priorities and ensuring timely delivery of commitments.
• Provide technical guidance and mentorship to junior team members when required.

Production Support

• Participate in a rotating production support roster.
• Troubleshoot production incidents and prioritize issue resolution within established SLA requirements.
• Conduct root cause analysis and implement preventive measures.
• Ensure platform stability, reliability, and operational excellence.

Experience

• Bachelor's Degree in Computer Science, Information Technology, Engineering, Data Science, or a related field.
• 8+ years of experience in Data Engineering, Data Warehousing, or Big Data technologies.
• Minimum 4+ years of hands-on Databricks experience in enterprise environments.
• Experience designing and implementing enterprise Data Warehouse solutions and modern Lakehouse architectures.

Why You’ll Love Working With Us

• Up to 24 Days Annual Leave
• Flexible & Hybrid Working
• Performance Bonus
• Enhanced EPF Contributions
• Training & Development (In-House & External)
• Travel Allowance & Expense Benefits
• Company Trips & Team Events
• Long Service Rewards

*OKU candidates with physical (mobility-related) disabilities or hearing impairment are encouraged to apply.*