Data Engineer

Experian; Cyberjaya; On-site; MYR 60,000 - 90,000; Full time

Job summary

A leading company in data analytics seeks an Analytics Engineer to join their Analytics Center of Excellence. The role involves managing large-scale data and developing client solutions using cutting-edge big data technologies. Ideal candidates will have strong coding skills, experience with data pipelines, and a solid understanding of machine learning principles.

Qualifications

• Experience in data warehouse technologies and ETL development.
• Strong coding skills in SQL and either Python, Java, or Scala.
• Experience with Big Data ML toolkits such as Mahout or SparkML.

Responsibilities

• Manage data at scale and work on key initiatives like Data Productizations.
• Build client solutions using various big data technologies.
• Create and maintain data pipelines for real-time and batch use cases.

Skills

Data modeling

ETL development

Data pipeline creation

SQL

Python

Machine learning

Big Data querying

Communication

Tools

Apache Spark

Airflow

AWS

Kafka

Hadoop

Job description

The role sits within our Decision Analytics, one of our four Global Business Lines.

Experian Decision Analytics helps clients achieve and sustain significant growth. We do this by enabling clients to make analytics-based customer decisions that support their strategic goals. As experts in uniting business understanding with consumer and business information, analytics, and strategy execution, we empower clients to optimize customer value and actively manage it over time. This role therefore has clear accountability for creating measurable value within our client organizations.

What you’ll be doing

In this role, you’ll be working as an Analytics Engineer within the Analytics Center of Excellence on Experian’s internal Cloud Platform. You’ll be managing data at scale and working on key initiatives such as Data Productizations, employing different big data technologies and front/back-end solutions to build client solutions.

More about you

• Experience in data warehouse technologies, data modeling, and ETL development.
• Experience in high-performing, large-scale, technology-driven environments.
• Exposure to creating and maintaining data pipelines (Apache Spark or similar) and workflow tools (Airflow or similar) for real-time and batch use cases. Cloud experience (AWS preferred).
• Basic understanding of machine learning, deep learning, wrappers, and APIs.
• Experience with Big Data ML toolkits such as Mahout, SparkML, or H2O (Mandatory).
• Strong coding skills in SQL and either Python, Java, or Scala.
• Exposure to database architecture using RDBMS or NoSQL: views, tables, disk usage, and relational diagrams.
• Excellent communication, interpersonal, and project management skills.
• Experience managing Hadoop clusters and resolving operational issues.
• Experience building stream-processing systems using solutions like Storm or Spark Streaming (a plus).
• Good knowledge of Big Data querying tools such as Pig, Hive, and Impala.
• Experience integrating data from multiple sources.
• Knowledge of ETL techniques and frameworks like Flume.
• Experience with messaging systems like Kafka or RabbitMQ.
• Understanding of Lambda Architecture, including its advantages and disadvantages.
• Experience with Cloudera, MapR, or Hortonworks distributions.