Data Engineer

INNOVATIVE CONSULTING PTE. LTD.; Singapore; On-site; SGD 60,000 - 90,000; Full time

Job summary

INNOVATIVE CONSULTING PTE. LTD. is hiring a Data Engineer to develop and maintain optimal data architectures and pipelines. You will solve complex data challenges, create robust ETL solutions, and mentor other professionals in best practices.

The ideal candidate has extensive data engineering experience, is proficient in programming with Python, and has knowledge of big data tools such as Hive and Spark. A collaborative approach and cloud service experience are highly valued.

Qualifications

• 3+ years of relevant experience in data engineering or analytics.
• Hands-on expertise with Python and SQL for data analysis.
• Knowledge of big data tools like Hive, Spark, and Hadoop.
• Experience with ETL processes and data governance.
• Familiarity with cloud computing services and big data solutions.

Responsibilities

• Design and maintain data architecture and pipelines for ETL.
• Solve complex data problems and deliver actionable insights.
• Code in Python and utilize Apache Spark for warehouse solutions.
• Collaborate across technology teams for smooth deployments.
• Mentor other data professionals on best practices.

Skills

Data engineering

Python

Ansible SQL

Apache Spark

ETL

Cloud services (AWS, Azure, GCP)

Data governance

Hadoop

Agile development

Collaboration and teamwork

Tools

Hive

Kafka

NoSQL

Job description

Job Description:

• Design, develop, optimize, and maintain data architecture and pipelines that adhere to ETL principles and business goals.
• Solve complex data problems to deliver insights that help the organization achieve its goals.
• Code in Python with tools like Apache Spark to build a multi-cluster data warehouse.
• Interact with other technology teams to define, prioritize, and ensure smooth deployments for other operational components.
• Advise, consult, mentor, and coach other data and analytics professionals on data standards and practices.
• Foster a culture of sharing, reuse, design for scale stability, and operational efficiency of data and analytical solutions.
• Codify best practices for future reuse in the form of accessible, reusable patterns, templates, and code bases to facilitate data capturing and management.
• Strong experience in mapping attributes, data profiling, data cleansing, and technical data quality etc.
• Strong experience in Ansi SQL and in-depth knowledge with structured, semi-structured & unstructured data.
• Must have good knowledge of data lake and working experience of migration projects in cloud with providers like AWS or Microsoft AZURE or GCP.
• Good to have experience in working with No SQL, Spark SQL using AWS Glue, EMR, and columnar data store.
• Good to understand data security features like data masking, data encryption, role based & fine grain access control mechanisms etc.

Qualifications:

• 3+ years of relevant experience in data engineering/analytics space.
• Expertise in SQL and data analysis and strong hands-on expertise with at least one programming language: Python.
• Strong knowledge in one or more of the following big data tools: Hive, Hadoop Impala, Spark, Kafka.
• Strong expertise in ETL, reporting tools, data governance, data warehousing, and hands-on experience.
• Experience developing solutions for cloud computing services and infrastructure.
• Experience developing and maintaining data warehouses in big data solutions.
• Up to date on industry trends within the analytics space from a data acquisition processing, engineering, and management perspective.
• Experience in agile development.
• Strong people skills, specifically in collaboration and teamwork.
• High level of curiosity, creativity, and problem-solving capabilities.