Data Engineer (AWS, Databricks)

NTT SINGAPORE PTE. LTD.; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

NTT Singapore PTE. LTD. is seeking a seasoned Cloud Operations professional to maintain data pipelines, optimize Spark jobs in Databricks, and manage cloud infrastructure with a focus on reliability and cost control.

The role requires hands-on AWS experience, Databricks expertise, and strong incident management skills to ensure high uptime and secure data environments across Enterprise workflows.

Qualifications

• Degree in Computer Science or Computer Engineering.
• Minimum 5 years working experience in system operations compliance and management areas.
• Project hands-on experience with AWS platform (primary requirement).
• Project experience in cloud operations or cloud architecture.
• Must be cloud certified (AWS).

Responsibilities

• Monitor and maintain production data pipelines to ensure 99.9% uptime and optimal performance.
• Implement comprehensive logging, alerting, and monitoring systems using Application monitoring tools.
• Perform regular health checks performance, job execution times, and resource utilization to identify and resolve bottlenecks proactively.
• Manage incident response procedures for pipeline failures, including root cause analysis, resolution, and post-incident reviews.
• Establish and maintain disaster recovery procedures and backup strategies for critical data assets within the Databricks environment.
• Conduct regular performance tuning of Spark jobs and Databricks cluster configurations to optimize cost and execution efficiency.
• Maintain comprehensive documentation for operational procedures, runbooks, and troubleshooting guides.
• Coordinate scheduled maintenance windows and system upgrades with minimal business impact.
• Manage user access controls, workspace configurations, and security policies within Application environments.

Skills

Databricks

Apache Spark

Data warehousing

Monitoring

Cloud cost control

DevOps

Stakeholder management

Education

Bachelors in CS/CE

AWS Certification

Tools

Databricks

AWS

IDMC

Tableau

Oracle DB

ML Ops in Databricks

SageMaker

DataRobot

Job description

Role and Responsibilities

Operational Responsibilities

• Monitor and maintain production data pipelines to ensure 99.9% uptime and optimal performance
• Implement comprehensive logging, alerting, and monitoring systems using Application monitoring tools
• Perform regular health checks performance, job execution times, and resource utilization to identify and resolve bottlenecks proactively
• Manage incident response procedures for pipeline failures, including root cause analysis, resolution, and post-incident reviews
• Establish and maintain disaster recovery procedures and backup strategies for critical data assets within the Databricks environment
• Conduct regular performance tuning of Spark jobs and Databricks cluster configurations to optimize cost and execution efficiency
• Maintain comprehensive documentation for operational procedures, runbooks, and troubleshooting guides
• Coordinate scheduled maintenance windows and system upgrades with minimal business impact
• Manage user access controls, workspace configurations, and security policies within Application environments

Requirements / Qualifications

Education & Experience

• Degree in Computer Science or Computer Engineering
• Minimum 5 years working experience in system operations compliance and management areas
• Project hands-on experience specifically with AWS platform (primary requirement)
• Project experience in cloud operations or cloud architecture
• Must be cloud certified (AWS)

Core Technical Skills

• Proficiency in Databricks platform, including workspace management, cluster configuration, and job orchestration
• Strong expertise in Apache Spark within Databricks environment, including Spark SQL, DataFrames, and RDDs
• Good in-depth understanding of data warehouse concepts, data profiling, data verification and advanced analytics techniques
• Strong knowledge of monitoring, incident management, and cloud cost control

Technology Stack Experience

• Databricks
• AWS cloud services and architecture
• IDMC (Informatica Data Management Cloud)
• Tableau for data visualization
• Oracle Database management
• ML Ops practices within Databricks environment
• STATA for statistical analysis is advantage
• Amazon SageMaker integration with Databricks
• DataRobot platform integration

Soft Skills & Stakeholder Management

• Good interpersonal skills with the ability to work with different groups of stakeholders
• Strong problem-solving skills and ability to work independently in a fast-paced environment with minimal supervision
• Excellent communication skills for technical documentation and cross team collaboration

Desirable Requirements

• AWS certification (Associate or Professional level) - highly preferred
• Exposure to hospital information/clinical systems is an added advantage
• Understanding of DevOps practices and CI/CD pipelines for Databricks based data engineering projects