lavori da Data Engineer in località Malaysia

Add expected salary to your profile for insights

We are hiring a Data Engineer to design, build, and operate batch and event-driven data pipelines on a modern on-premise data platform.

This role focuses on data ingestion, transformation, and processing, using Apache Spark and Kafka, supporting analytics, reporting, and operational dashboards. You will work closely with Platform Integration Engineers, who manage the underlying infrastructure and streaming platform.

Apache Spark (SQL / PySpark / Structured Processing), Apache Kafka, Batch & Streaming Data Pipelines, ETL / ELT, CDC (Change Data Capture), PostgreSQL / Relational Databases, Docker / Kubernetes, Linux, On-Prem Data Platform

• Design, build, and operate large-scale batch processing pipelines using Apache Spark
• Develop ETL / ELT workflows for analytics and reporting
• Implement data ingestion pipelines from databases, APIs, and event sources
• Optimise data pipelines for performance, reliability, resource efficiency and data quality

• Consume and process events from Kafka
• Introduce streaming or near-real-time processing when required
• Work with CDC pipelines and event-based data sources
• Support schema evolution and downstream data consumption

• Collaborate with Platform Integration Engineers on deployment, observability, and reliability
• Work with backend and application teams on data requirements and integration
• Participate in troubleshooting data pipeline issues and performance bottlenecks
• Maintain documentation, data models, and design decisions

We do not expect every candidate to meet every requirement, but strong experience in most of the following is important.

• Strong programming skills in Python, Java, or Scala
• Hands-on experience with Apache Spark
• Working knowledge of Apache Kafka or other messaging platforms
• Strong SQL skills and experience with PostgreSQL or other RDBMS
• Familiarity with Linux environments

• Experience with Apache Flink, Spark Streaming, Kafka Streams
• Exposure to CDC tools (Debezium, Maxwell, GoldenGate)
• Experience with Docker and Kubernetes
• Familiarity with workflow orchestration tools (Airflow or similar)
• Experience with monitoring, CI/CD, or infrastructure-as-code tools

• Data Engineers who enjoy pipeline design and optimisation
• Backend engineers moving into data engineering
• Candidates interested in batch and streaming data systems
• Strong fundamentals with willingness to learn are welcome

• Ownership of core data pipelines in a production system
• Hands-on experience with Spark, Kafka, and distributed data systems
• Clear growth path toward Senior Data Engineer / Data Architect
• High-impact role in a small, engineering-focused team