Lead Data Engineer

Temus; Singapore; On-site; SGD 80,000 - 120,000; Full time

Job summary

A leading digital transformation firm in Singapore is seeking an experienced Data Engineer to develop and implement data storage solutions using AWS and Databricks. The role involves creating data pipelines, ensuring data consistency, and integrating various data sources while following best practices in security and compliance. Strong knowledge of AWS, Databricks, and Proficiency in programming languages is essential. The ideal candidate will have excellent problem-solving skills and a collaborative mindset. This opportunity offers competitive benefits and a diverse work environment.

Qualifications

• 5+ years of experience in data engineering with AWS and Databricks expertise.
• Proficiency in Python, Java, or Scala for building data pipelines.
• Strong knowledge of SQL and NoSQL databases.

Responsibilities

• Design and architect data storage solutions using AWS services.
• Create, manage, and optimize data pipelines for data processing.
• Implement security best practices and data encryption methods.

Skills

AWS services

Databricks

Informatica IDMC

Python

SQL

NoSQL databases

Data modeling

Problem-solving

Collaboration

Education

Bachelor’s or master’s degree in computer science, data engineering, or a related field

Tools

Apache Spark

Hadoop

Tableau

Power BI

Docker

Kubernetes

Job description

Temus was established by Temasek in partnership with UST, to provide digital transformation solutions for the private and public sectors as we aspire to be a strategic partner in realising the Singapore Government’s Smart Nation vision. We are headquartered in Singapore and have more than 400 employees across a wide range of disciplines in strategy, design, architecture, technology, data & AI.

Roles And Responsibilities:

• Design and architect data storage solutions, including databases, data lakes, and warehouses, using AWS services such as Amazon S3, Amazon RDS, Amazon Redshift, and Amazon DynamoDB, along with Databricks' Delta Lake. Integrate Informatica IDMC for metadata management and data cataloging.
• Create, manage, and optimize data pipelines for ingesting, processing, and transforming data using AWS services like AWS Glue, AWS Data Pipeline, and AWS Lambda, Databricks for advanced data processing, and Informatica IDMC for data integration and quality.
• Integrate data from various sources, both internal and external, into AWS and Databricks environments, ensuring data consistency and quality, while leveraging Informatica IDMC for data integration, transformation, and governance.
• Develop ETL (Extract, Transform, Load) processes to cleanse, transform, and enrich data, making it suitable for analytical purposes using Databricks' Spark capabilities and Informatica IDMC for data transformation and quality.
• Monitor and optimize data processing and query performance in both AWS and Databricks environments, making necessary adjustments to meet performance and scalability requirements. Utilize Informatica IDMC for optimizing data workflows.
• Implement security best practices and data encryption methods to protect sensitive data in both AWS and Databricks, while ensuring compliance with data privacy regulations. Employ Informatica IDMC for data governance and compliance.
• Implement automation for routine tasks, such as data ingestion, transformation, and monitoring, using AWS services like AWS Step Functions, AWS Lambda, Databricks Jobs, and Informatica IDMC for workflow automation.
• Maintain clear and comprehensive documentation of data infrastructure, pipelines, and configurations in both AWS and Databricks environments, with metadata management facilitated by Informatica IDMC.
• Collaborate with cross-functional teams, including data scientists, analysts, and software engineers, to understand data requirements and deliver appropriate solutions across AWS, Databricks, and Informatica IDMC.
• Identify and resolve data-related issues and provide support to ensure data availability and integrity in both AWS, Databricks, and Informatica IDMC environments.
• Optimize AWS, Databricks, and Informatica resource usage to control costs while meeting performance and scalability requirements. • Stay up-to-date with AWS, Databricks, Informatica IDMC services, and data engineering best practices to recommend and implement new technologies and techniques.

Requirements:

• Bachelor’s or master’s degree in computer science, data engineering, or a related field.
• Minimum 5 years of experience in data engineering, with expertise in AWS services, Databricks, and/or Informatica IDMC.
• Proficiency in programming languages such as Python, Java, or Scala for building data pipelines.
• Evaluate potential technical solutions and make recommendations to resolve data issues especially on performance assessment for complex data transformations and long running data processes.
• Strong knowledge of SQL and NoSQL databases.
• Familiarity with data modeling and schema design.
• Excellent problem-solving and analytical skills.
• Strong communication and collaboration skills.
• AWS certifications (e.g., AWS Certified Data Analytics - Specialty, AWS Certified Data Analytics - Specialty), Databricks certifications, and Informatica certifications are a plus.

Preferred Skills:

• Experience with big data technologies like Apache Spark and Hadoop on Databricks.
• Knowledge of containerization and orchestration tools like Docker and Kubernetes. • Familiarity with data visualization tools like Tableau or Power BI.
• Understanding of DevOps principles for managing and deploying data pipelines. • Experience with version control systems (e.g., Git) and CI/CD pipelines.
• Knowledge of data governance and data cataloguing tools, especially Informatica IDMC.

Temus is an equal opportunities employer. We welcome applications from all. We do not discriminate by race, religion, belief, ethnicity, origin, disability, age, partnership status, sexual orientation, or gender identity.

We see the diversity of our team as a strategic advantage, and we work actively to maintain it.

Interested in building your career at Temus? Get future opportunities sent straight to your email.

Accepted file types: pdf, doc, docx, txt, rtf

LinkedIn Profile

Kindly share your LinkedIn Profile link.