Data Architect -Data brick and AWS

PATH INFOTECH PTE. LTD.; Singapore; On-site; SGD 90,000 - 150,000; Full time

Job summary

PATH INFOTECH PTE. LTD. is seeking an experienced data engineer to manage data storage and analytics platforms across AWS and Databricks. You will optimize databases, data lakes, and warehouses while leveraging Delta Lake for processing and Informatica IDMC for metadata and governance.

You will design robust ETL pipelines, ingest and transform data, and ensure data quality, security, and compliance. The role emphasises scalable architectures and automated workflows.

Qualifications

• Experience designing and managing data storage solutions on AWS (S3, RDS, Redshift, DynamoDB).
• Hands-on use of Databricks and Delta Lake for data processing and analytics.
• Familiarity with Informatica IDMC for metadata, cataloging, and governance.

Responsibilities

• Manage and optimise data storage across databases, data lakes, and warehouses using AWS services and Databricks.
• Design and tune data pipelines for ingest, processing, and transformation with AWS Glue, Data Pipeline, Lambda and Databricks.
• Integrate data from multiple sources into AWS/Databricks with Informatica IDMC for quality and governance.
• Oversee ETL processes to cleanse, transform, and enrich data for analytics using Databricks Spark and IDMC.
• Monitor performance and security, optimise workloads, and ensure compliance with data privacy regulations.

Skills

ETL

Data integration

Data governance

Data quality

Data security

Spark

Cloud architecture

Tools

Amazon S3

Amazon RDS

Amazon Redshift

Amazon DynamoDB

Databricks Delta Lake

Informatica IDMC

AWS Glue

AWS Data Pipeline

AWS Lambda

Databricks

Spark

Job description

Roles And Responsibilities:

• Manage and Optimise existing data storage solutions, including databases, data lakes, and warehouses, using AWS services such as Amazon S3, Amazon RDS, Amazon Redshift, and Amazon DynamoDB, along with Databricks' Delta Lake. Integrate Informatica IDMC for metadata management and data cataloging.
• Manage and optimize data pipelines for ingesting, processing, and transforming data using AWS services like AWS Glue, AWS Data Pipeline, and AWS Lambda, Databricks for advanced data processing, and Informatica IDMC for data integration and quality.
• Integrate data from various sources, both internal and external, into AWS and Databricks environments, ensuring data consistency and quality, while leveraging Informatica IDMC for data integration, transformation, and governance.
• Manage ETL (Extract, Transform, Load) processes to cleanse, transform, and enrich data, making it suitable for analytical purposes using Databricks' Spark capabilities and Informatica IDMC for data transformation and quality.
• Monitor and optimize data processing and query performance in both AWS and Databricks environments, making necessary adjustments to meet performance and scalability requirements. Utilize Informatica IDMC for optimizing data workflows.
• Implement security best practices and data encryption methods to protect sensitive data in both AWS and Databricks, while ensuring compliance with data privacy regulations. Employ Informatica IDMC for data governance and compliance.
• Implement automation for routine tasks, such as data ingestion, transformation, and monitoring, using AWS services like AWS Step Functions, AWS Lambda, Databricks Jobs, and Informatica IDMC for workflow automation.