Senior Data Engineer

Software International Corporation; Kuala Lumpur; Hybrid; MYR 180,000 - 260,000; Full time

Benefits offered by this job

Up to 24 Days Annual Leave

Flexible & Hybrid Working

Performance Bonus

Enhanced EPF Contributions

Training & Development (In-House &/or)

Travel Allowance & Expense Benefits

Company Trips & Team Events

Long Service Rewards

Job summary

Software International Corporation in Kuala Lumpur is seeking an experienced Data Engineer to design and maintain scalable data pipelines, data warehouse/lakehouse architectures, and governance-enabled platforms. The role emphasizes Databricks, PySpark, and modern data models for analytics and AI/ML use cases.

You will implement CI/CD for Databricks, collaborate with stakeholders, and mentor junior engineers while ensuring data quality, security, and performance across the platform.

Qualifications

• Bachelor’s degree in a relevant field and 8+ years in data engineering or big data.
• Hands-on Databricks experience in enterprise environments.
• Experience designing enterprise Data Warehouse and Lakehouse architectures.
• Strong communication and stakeholder collaboration skills.
• Experience in governance, data security, and CI/CD practices.

Responsibilities

• Design, develop, and maintain scalable data pipelines and data products using Databricks and PySpark.
• Build and optimize ETL/ELT solutions supporting batch and near real-time data processing.
• Design, develop, and maintain enterprise Data Warehouse and Lakehouse solutions for reporting and analytics.
• Develop enterprise data models using Data Lake and Delta Tables; implement star/snowflake schemas and SCDs.
• Implement CI/CD pipelines for Databricks; integrate with GitHub Actions and DevOps processes.
• Engage with business users and data consumers to gather requirements and deliver data solutions.
• Provide technical guidance and mentorship to junior team members; ensure platform stability and performance.

Skills

Data engineering

Stakeholder engagement

Mentorship

Problem solving

Education

Bachelor's Degree in Computer Science, Information Technology, Engineering, Data Science, or a related field

Tools

Databricks

GitHub

GitHub Actions

CI/CD

Unity Catalog

Job description

• Design, develop, and maintain scalable data pipelines and data products using Databricks and PySpark.
• Build and optimize ETL/ELT solutions supporting batch and near real-time data processing.
• Design, develop, and maintain enterprise Data Warehouse and Lakehouse solutions that support reporting, analytics, and AI/ML use cases.
• Develop and maintain enterprise data models using Data Lake and Delta Tables.
• Design and implement dimensional data models, including Fact and Dimension tables, Star Schema, Snowflake Schema, and Slowly Changing Dimensions (SCD).
• Ensure data quality, scalability, security, and performance across the data platform.
• Implement best practices for code management, testing, deployment, and operational monitoring.

Databricks Platform & Governance

• Implement and manage Unity Catalog for centralized governance, data discovery, and security.
• Design and enforce governance frameworks including:
• Role-Based Access Control (RBAC)
• Attribute-Based Access Control (ABAC)
• Fine-grained data permissions
• Data lineage and auditing
• Configure and manage Data Sharing to support secure external and internal data collaboration.
• Support the adoption and utilization of Databricks Genie, including:
• Security and access governance controls

Performance Optimization

• Perform advanced PySpark performance tuning and troubleshooting.
• Optimize query performance, cluster utilization, partitioning strategies, and workload management.
• Identify bottlenecks and proactively improve platform efficiency and cost optimization.
• Optimize Data Warehouse and Lakehouse workloads to support high-performance reporting and analytics processing.

DevOps & Automation

• Design and implement CI/CD pipelines for Databricks solutions.
• Integrate Databricks development lifecycle with GitHub, GitHub Actions, and enterprise DevOps processes.
• Automate deployment, testing, code validation, and release management processes.
• Establish infrastructure and data engineering best practices.

Stakeholder Management

• Engage with business users, data consumers, architects, analysts, and technology leadership to gather requirements and deliver data solutions.
• Translate business requirements into scalable technical designs, data models, and platform capabilities.
• Communicate effectively with stakeholders across multiple organizational levels.
• Work independently while managing priorities and ensuring timely delivery of commitments.
• Provide technical guidance and mentorship to junior team members when required.

Production Support

• Participate in a rotating production support roster.
• Troubleshoot production incidents and prioritize issue resolution within established SLA requirements.
• Conduct root cause analysis and implement preventive measures.
• Ensure platform stability, reliability, and operational excellence.

Experience

• Bachelor's Degree in Computer Science, Information Technology, Engineering, Data Science, or a related field.
• 8+ years of experience in Data Engineering, Data Warehousing, or Big Data technologies.
• Minimum 4+ years of hands-on Databricks experience in enterprise environments.
• Experience designing and implementing enterprise Data Warehouse solutions and modern Lakehouse architectures.

Why You’ll Love Working With Us

• Up to 24 Days Annual Leave
• Flexible & Hybrid Working
• Performance Bonus
• Enhanced EPF Contributions
• Training & Development (In-House & External)
• Travel Allowance & Expense Benefits
• Company Trips & Team Events
• Long Service Rewards

*OKU candidates with physical (mobility-related) disabilities or hearing impairment are encouraged to apply.*