Software Engineer (Apache / Pyspark / Python) - Contract Role

QUESS ENGINEERING PTE. LTD.

Singapore

On-site

SGD 120,000 - 180,000

Full time

8 days ago

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

QUESS ENGINEERING PTE. LTD. is seeking an experienced Big Data/Data Engineering professional in Singapore to design, build, and optimize data workflows using Hadoop/Cloudera and Spark.

You will develop Python Django-based microservices and leverage Redis, Hive, and Kafka for reliable data processing. The role requires deep expertise in data warehouse management, streaming analytics, and collaboration with data scientists to drive business initiatives.

Qualifications

  • 7+ years of experience in Big Data and data engineering roles.
  • Proficiency with Cloudera, Hadoop, Hive, and MapReduce.
  • Advanced knowledge of Python, Bash, and Perl.
  • Strong skills in PySpark, Impala, and Kafka.
  • Hands-on experience with data warehouse management and optimization.
  • Experience with MongoDB and relational databases.
  • Proficiency with Apache Airflow or similar orchestration tools.
  • Knowledge of Linux environments and other DevOps tools.
  • Familiarity with cloud platforms such as AWS, Azure, or GCP.
  • Experience with real-time data processing and streaming.

Responsibilities

  • Design, develop, and optimize data workflows and pipelines using Hadoop, Cloudera, and Spark.
  • Develop microservice APIs using Python Django and Redis as a data store.
  • Develop datawarehouse applications on Mainframe with Teradata.
  • Manage data storage in data warehouse solutions including Cloudera and Hive.
  • Create a unified execution framework to oversee vCPU/memory for YARN apps and logging.
  • Implement data processing workflows with PySpark, Hive, MapReduce, and Impala.
  • Build robust data ingestion frameworks with Kafka and ETL tools.
  • Work with structured and unstructured data using Python, Bash, and Perl scripts.
  • Manage MongoDB and ensure integration with other data systems.
  • Address performance issues like Data Spill, Skew, and resource constraints on Spark.
  • Use Apache Airflow to schedule, manage, and monitor ETL workflows.
  • Collaborate with data scientists, analysts, and engineers to meet business needs.

Skills

Hadoop
Cloudera
Spark
Python
Django
Redis
Teradata
Hive
MapReduce
Impala
Kafka
PySpark
Airflow
MongoDB
Linux
AWS
Azure
GCP
Bash
Perl
ETL
Data Warehousing

Education

Bachelor's or Master's in CS/IT

Tools

Airflow
SQL
MongoDB
Teradata

Job description

Key Responsibilities
  • Design, develop, and optimize data workflows and pipelines using Apache Hadoop, Cloudera, and Spark.
  • Develop microservice APIs using Python Django framework and utilized Redis as a data store
  • Developed various datawarehouse applications on Mainframe with Teradata.
  • Manage and optimize data storage in data warehouse solutions, including Cloudera and Hive.
  • Develop a unified execution framework to oversee vCPU and memory configuration for YARN applications and logging processes.
  • Implement and manage data processing workflows using tools like PySpark, Hive, MapReduce, and Impala.
  • Build robust data ingestion frameworks with Kafka and other ETL tools, ensuring data quality and reliability.
  • Work with structured and unstructured data using Python, Bash, and Perl scripting to analyze, process, and transform data.
  • Manage MongoDB and ensure integration with other data storage and processing systems.
  • Identify and address performance degradation issues such as Data Spill, Skew, and resource constraints on Spark Batch & Streaming applications in production environments.
  • Use Apache Airflow to schedule, manage, and monitor ETL workflows.
  • Work closely with data scientists, analysts, and other engineers to meet business requirements and drive data initiatives.
Requirements:
  • Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.
  • 7+ years of experience in Big Data and data engineering roles.
  • Proficiency with Cloudera, Hadoop, Hive, and MapReduce.
  • Advanced knowledge of Python, Bash, and Perl.
  • Strong skills in PySpark, Impala, and Kafka.
  • Hands-on experience with data warehouse management and optimization.
  • Experience with MongoDB and relational databases.
  • Proficiency with Apache Airflow or similar orchestration tools.
  • Knowledge of Apache, Linux environments, and other DevOps tools.
  • Familiarity with cloud platforms such as AWS, Azure, or GCP.
  • Experience with real-time data processing and streaming.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Applications Developer
Applications Developer

RISKDATA CONSULTING PTE. LTD. • Singapore

On-site
SGD 120,000 - 180,000
Data Engineer
Data Engineer

ZENITH INFOTECH (S) PTE LTD. • Singapore

On-site
SGD 120,000 - 180,000
Applications Developer
Applications Developer

REGTECH INSIGHT PTE. LTD. • Singapore

On-site
SGD 120,000 - 180,000
BigData Engineer
BigData Engineer

NTT Data Singapore • Singapore

On-site
SGD 95,000 - 150,000
Applications Developer
Applications Developer

UNISONEDGE CONSULTING PTE. LTD. • Singapore

On-site
SGD 90,000 - 130,000
APPLICATIONS DEVELOPER
APPLICATIONS DEVELOPER

Riskdata Consulting • Singapore

On-site
SGD 120,000 - 180,000
APPLICATIONS DEVELOPER
APPLICATIONS DEVELOPER

UARROW PTE. LTD. • Singapore

On-site
SGD 120,000 - 180,000
APPLICATIONS DEVELOPER
APPLICATIONS DEVELOPER

JEET ANALYTICS PTE. LTD. • Singapore

On-site
SGD 90,000 - 130,000
Data Engineer
Data Engineer

ARYAN SOLUTIONS PTE. LTD. • Singapore

On-site
SGD 90,000 - 150,000
Solution Architect
Solution Architect

UNISYNC SYSTEMS PTE. LTD. • Singapore

On-site
SGD 180,000 - 240,000