Senior Data Engineer - AWS/Databricks & Data Security Focus

Exasoft Pte. Ltd.; Singapore; On-site; SGD 120,000 - 170,000; Full time

Job summary

Exasoft Pte. Ltd. is seeking a senior Data Engineer to design scalable data storage and pipelines on AWS and Databricks. You will implement data governance with Informatica IDMC and build robust ETL processes for analytics.

Ideal candidates have 5+ years in data engineering, strong Python, SQL/NoSQL skills, and experience with Docker/Kubernetes. You will collaborate with data scientists and engineers to optimize performance and ensure data quality while controlling costs.

Qualifications

• Bachelor's or Master's degree in computer science, data engineering, or a related field.
• Minimum 5+ years of experience in data engineering with expertise in AWS, Databricks, and/or Informatica IDMC.
• Proficiency in Python for building data pipelines.
• Strong SQL and NoSQL database knowledge.
• Experience with data governance and data cataloguing tools, especially Informatica IDMC.
• Familiarity with data visualization tools like Tableau or Power BI.
• Familiarity with Docker and Kubernetes.

Responsibilities

• Design data storage solutions using AWS services such as S3, RDS and Databricks Delta Lake.
• Create, manage and optimize data pipelines for ingestion, processing, and transformation.
• Integrate data from multiple sources into AWS and Databricks, ensuring quality and governance.
• Develop ETL processes to cleanse, transform, and enrich data for analytics.
• Monitor and optimize data processing and query performance in AWS and Databricks.
• Implement security best practices and data encryption; ensure privacy compliance.
• Automate routine tasks with AWS Step Functions, AWS Lambda, Databricks Jobs and Informatica IDMC.
• Maintain documentation with metadata management via Informatica IDMC.
• Collaborate with data scientists, analysts, and engineers.
• Identify and resolve data-related issues and ensure data availability.
• Optimize resource usage to control costs.
• Stay up-to-date with AWS, Databricks and Informatica IDMC services and best practices.

Skills

Python

SQL

NoSQL databases

Data modelling

Problem solving

Communication

Collaboration

Education

Bachelor's or Master's degree in Computer Science or Data Engineering

Tools

Docker

Kubernetes

Tableau/Power BI

AWS

Databricks

Informatica IDMC

Job description

Exasoft Pte. Ltd. is seeking a senior Data Engineer to design scalable data storage and pipelines on AWS and Databricks. You will implement data governance with Informatica IDMC and build robust ETL processes for analytics.

Ideal candidates have 5+ years in data engineering, strong Python, SQL/NoSQL skills, and experience with Docker/Kubernetes. You will collaborate with data scientists and engineers to optimize performance and ensure data quality while controlling costs.