Senior Python Developer

ENVISION DIGITAL TECHNOLOGIES SDN BHD; Kuala Lumpur; Hybrid; MYR 150,000 - 200,000; Full time

Job summary

A technology solutions firm in Kuala Lumpur seeks a Senior Python Developer to design and optimize ETL pipelines for large-scale data processing. The ideal candidate should have substantial experience in Python and Django, as well as expertise in handling high-volume datasets. Responsibilities include developing back-end services and integrating APIs for data collection. This role offers a dynamic work culture and flexible arrangements, emphasizing continuous improvement and collaboration.

Benefits

13th Month Salary (AWS)

Flexi Allowance

Insurance, Medical, Dental & Optical Coverage

Parking & Transport Support

Hybrid Work Arrangement

Office Near LRT

Dynamic & Supportive Environment

Qualifications

• More than 2 years of professional experience in Python, focusing on data processing.
• Hands-on experience with Scrapy or similar frameworks for ETL systems.
• Excellent communication skills for stakeholder coordination.

Responsibilities

• Design and develop ETL pipelines using Python and Scrapy.
• Integrate external APIs and develop backend services using Django and Spring Boot.
• Monitor and optimize pipeline performance with automated alerts.

Skills

Python

Django

Java Spring Boot

RESTful API integration

SQL (MySQL, PostgreSQL)

Scrapy

Data validation

Communication

Tools

Linux

Docker

Airflow

Job description

Add expected salary to your profile for insights

We are looking for a Senior Python Developer to design, implement, and optimize large-scale data ingestion, processing, and reporting pipelines.

The ideal candidate will have strong experience in Python, Django, and Java Spring Boot, with a proven ability to handle high-volume datasets (up to billions of transactions) efficiently.

This role also requires strong communication skills to coordinate with internal teams and external partners for data collection, system integration, and continuous improvement of data workflows.

Key Responsibilities

• Design and develop ETL pipelines using Python and Scrapy for multi-source data ingestion, transformation, and validation at scale.
• Integrate external APIs for structured data collection, including authentication, pagination, rate limiting, and error handling.
• Develop backend services using Django (Python) and Spring Boot (Java) to process, store, and automate large datasets.
• Handle and optimize large-volume data processing, ensuring scalability, reliability, and high performance (e.g., handling 1 billion+ transaction records).
• Implement automated reporting pipelines to generate and distribute accurate operational and analytical reports.
• Verify and maintain data integrity, ensuring accuracy, completeness, and consistency across systems.
• Monitor and optimize pipeline performance, including throughput, latency, and health metrics, with automated alerts.
• Automate data verification and reconciliation workflows for anomaly detection and quality assurance.
• Collaborate with various parties — vendors, internal technical teams, and business stakeholders — to collect information, clarify requirements, and resolve data issues.
• Document workflows and technical processes for data validation, transformation, and reporting.

Requirements

• More than 2 years of professional experience in Python, Django development, focusing on data processing and ETL systems.
• Strong programming experience in Java Spring Boot for backend and API development.
• Hands‑on experience with Scrapy or similar frameworks for large-scale data ingestion.
• Proven experience processing and optimizing very large datasets (hundreds of millions to billions of records).
• Proficient in RESTful API integration and advanced request handling (authentication, pagination, retries).
• Strong knowledge of SQL (MySQL, PostgreSQL, or ClickHouse) and data modeling.
• Experience generating data reports using Python (e.g., Pandas, Matplotlib) or BI/dashboard tools.
• Familiarity with Linux, Docker, and scheduling systems (Airflow, Cronicle, cron, etc).
• Excellent communication and collaboration skills to coordinate with multiple internal and external stakeholders.

Nice to Have

• Experience with Kafka, Redis, or other message queues for high-throughput pipelines.
• Familiarity with object storage systems (MinIO, S3) and analytical databases (ClickHouse, StarRocks).
• Experience with monitoring and observability (Prometheus, Grafana, Sentry).
• Exposure to reporting tools like Metabase or Apache Superset.
• Multilingual proficiency (English and Chinese) for regional coordination.

What Success Looks Like

• Stable, high-performance ETL and reporting pipelines capable of handling 1 billion+ records.
• Automated and transparent data reporting that supports fast decision-making.
• Efficient data validation workflows and clear documentation.
• Smooth communication and collaboration across internal and external teams.
• Continuous improvements in data reliability and operational efficiency.

Unlock job insights

Salary match Number of applicants Skills match

Your application will include the following questions:

• What's your expected monthly basic salary?
• Which of the following types of qualifications do you have?
• How many years' experience do you have as a Python Developer?
• Which of the following programming languages are you experienced in?
• How would you rate your Mandarin language skills?
• Which of the following languages are you fluent in?

Envision Digital is a technology company specializing in delivering custom software development and digital solutions. We focus on building scalable systems, providing enterprise-level support, and fostering innovation through cloud technologies and continuous improvement. Our lowercase brand reflects our belief in continuous growth and agility — a reminder to keep learning, adapting, and improving. We embrace experimentation, value collaboration, and are committed to delivering quality outcomes that exceed expectations. At envision digital, we grow with our people and our clients.

Perks and benefits 🎁 13th Month Salary (AWS), Flexi Allowance 🩺 Insurance, Medical, Dental & Optical Coverage 🚗 Parking & Transport Support 🏠 Hybrid Work Arrangement 🚆 Office Near LRT 🤝 Modern Work Culture 🌟 Dynamic & Supportive Environment