Senior Data Engineer (AWS & Databricks) | Contract

nexbridge recruitment pte. ltd.; Singapore; Presencial; SGD 90.000 - 180.000; Jornada completa

Descripción de la vacante

nexbridge recruitment pte. ltd. is seeking an experienced Data Engineer to design, build and maintain scalable batch and streaming pipelines on AWS and Databricks.

You will contribute to lakehouse architectures with Delta Lake and Spark, ensure data quality and governance, and collaborate with data scientists and stakeholders. The role requires strong Python/SQL skills, 2+ years on cloud platforms, and experience with CI/CD and Terraform.

Formación

• Degree in Computer Science, Software Engineering, Data Engineering, Information Systems or a related field.

Responsabilidades

• Design, develop and maintain scalable batch and streaming data pipelines across AWS and Databricks.

Descripción del empleo

Key Responsibilities

• Design, develop and maintain scalable batch and streaming data pipelines across AWS and Databricks.
• Build and manage lakehouse architectures using Apache Spark, PySpark and Delta Lake, including Bronze, Silver and Gold data layers.
• Integrate data from databases, APIs, files and event streams using incremental ingestion and change data capture.
• Implement data quality checks, schema validation, reconciliation, metadata management and data lineage.
• Develop automated workflows and CI/CD pipelines using Databricks Workflows, AWS services, Git and infrastructure-as-code tools.
• Monitor pipeline performance, troubleshoot issues and implement logging, alerts and recovery procedures.
• Optimise Spark and SQL workloads for performance, reliability and cloud cost efficiency.
• Implement data security, access controls, encryption and governance in line with agency requirements.
• Establish engineering standards, review code and architecture, and mentor junior engineers.
• Collaborate with data scientists, analysts, architects and business stakeholders to deliver reliable data solutions.
• Maintain technical documentation, operating procedures and knowledge-transfer materials.

Requirements & Qualifications

• Degree in Computer Science, Software Engineering, Data Engineering, Information Systems or a related field.
• 6-10 years of relevant data or software engineering experience, including at least 2 years working with cloud-based data platforms.
• Strong hands-on experience in Python, PySpark, SQL, Apache Spark and Delta Lake.
• Proven experience building and operating production data pipelines using Databricks and AWS.
• Good knowledge of AWS services such as S3, Glue, Lake Formation, IAM, Lambda, Redshift and CloudWatch.
• Experience with lakehouse architecture, data modelling, incremental processing, change data capture and data quality management.
• Familiarity with workflow orchestration, CI/CD, Git and infrastructure-as-code tools such as Terraform.
• Experience with pipeline monitoring, troubleshooting, performance optimisation and cloud security.
• Ability to lead technical design, conduct code reviews, establish engineering standards and mentor engineers.
• Strong problem-solving, communication and stakeholder management skills.

Preferred Skills

• Experience with Unity Catalog, Databricks Workflows, Lakeflow, Delta Live Tables or Structured Streaming.
• Familiarity with Airflow, dbt, Kafka, AWS DMS, MSK or Kinesis.
• Experience building data products, data contracts, feature engineering or ML data pipelines.
• Knowledge of Singapore public-sector data governance, security and compliance requirements.
• Relevant Databricks or AWS certifications.