Data Engineer

Avinity Analytics; Selangor; On-site; MYR 80,000 - 120,000; Full time

Job summary

A data analytics firm located in Malaysia, Selangor is looking for a skilled data engineer. As part of the team, you will design and maintain optimal data pipeline architecture, wrangle large data sets, and work with various stakeholders on data-related technical issues. The ideal candidate has a Bachelor's degree in Computer Science or a related field, is proficient in Python and SQL, and has experience with data engineering tools like Airflow, Hadoop, and cloud data platforms. This role offers exciting opportunities to shape data-driven decision-making.

Qualifications

• Bachelor's degree or higher in Computer Science, Engineering, Information Technology or related field.
• Proficiency in scripting languages such as Python and SQL.
• Experience with data engineering tools like Hadoop and Spark.

Responsibilities

• Design and maintain optimal data pipeline architecture.
• Work with stakeholders to assist with data-related technical issues.
• Implement data security and governance best practices.

Skills

Python

SQL

Data modeling

Data warehousing

Data governance

Education

Bachelor's degree in Computer Science

Tools

Airflow

Hadoop

Snowflake

BigQuery

Job description

Responsibilities

• Design and maintain optimal data pipeline architecture via scheduling and authoring tools such as Airflow & dbt
• Wrangling large, complex data sets & modelling data to meet functional / non-functional business requirements.
• Optimize data storage and query processing for performance and scalability.
• Work with stakeholders including the Executive, Product, Data and Design teams to assist with data-related technical issues and support their data infrastructure needs.
• Implement data security and governance best practices.

Qualifications

• Bachelor's degree or higher in Computer Science, Engineering, Information Technology or other related field.
• Proficiency in scripting languages - Python, SQL
• Good understanding of relational DB, data structures, models, data warehouses & data lakes
• Familiarity with databases & data warehouses - MySQL, PostgreSQL, MSSQL, NoSQL, Snowflake, BigQuery
• [+] Experience in data engineering tools & languages - Hadoop, Spark, Kafka, Airflow
• [+] Experience with cloud-based data platforms (e.g., AWS, Azure)
• [+] Experience in data modelling (dbt) , ML modelling & MLOps