Staff Software Engineer (Platform Engineering), National Environment Agency

GovTech Singapore; Singapore; Hybrid; SGD 120,000 - 180,000; Full time

Benefits offered by this job

Flexible work arrangements

Learning & development opportunities

Job summary

GovTech Singapore is seeking a Senior Data & Platform Engineer to lead the Weather Data Lake build on Databricks, replacing the legacy system and serving as the single source of meteorological data for MSS.

You will own the MSS data platform architecture on GCC 2.0 Azure, design production pipelines, and mentor junior engineers while collaborating with MSS to meet the CDMS deadline and drive governance across the lake.

Qualifications

• 8+ years in data engineering, platform engineering, or backend software engineering, with at least 3 years in a senior or lead capacity.
• Strong hands-on experience with Databricks (Delta Lake, Unity Catalog, Structured Streaming, Workflows) and the lakehouse architecture pattern.
• Deep familiarity with Azure — ADLSv2, Blob Storage, Data Factory, and networking fundamentals (VNet, Private Endpoints, ExpressRoute).
• Strong Terraform experience for data platform infrastructure: provisioning Databricks workspaces, storage, networking, and IAM in a modular, version-controlled manner.
• Proficiency in Python and SQL. Solid hands-on Spark (PySpark or Scala) experience building and optimising large-scale data processing on Databricks.
• Demonstrated ability to make platform architecture decisions, write ADRs, and present trade-offs to technical and non-technical stakeholders.
• Track record of mentoring junior engineers and raising the engineering bar across a team.
• Experience with data governance frameworks, data quality tooling, and lineage tracking in production.
• Experience in GCC 2.0 or equivalent regulated cloud environments with strict change control is highly desirable.
• Strong communication skills bridging data platform concepts with domain experts, product managers, and leadership.

Responsibilities

• Design and build the Weather Data Lake on Databricks (Unity Catalog, Delta Lake, Structured Streaming) running on ADLSv2 within GCC 2.0. Make and document platform architecture decisions covering compute, storage tiering, governance, and cost management.
• Architect production data pipelines ingesting from GTS, ADES, AirViro, IWXXM, and WIS 2.0 sources. Build these using Spark (PySpark) on Databricks with CDC patterns and full data lineage.
• Own the data quality framework. Implement QC pipelines serving both the real-time forecasting path and the post-correction analytics path from a common Azure Blob landing zone.
• Own the infrastructure as code layer, defining all platform resources (Databricks workspaces, storage, networking, IAM) in Terraform with modular configurations. Establish staging and production environment separation with appropriate retention policies and CI/CD pipelines.
• Define integration contracts between the Azure Landing Zone (ingestion) and the Weather Data Lake (analytics), ensuring operational systems do not take direct dependencies on the lake.
• Mentor junior data engineers and software engineers. Conduct code reviews, run architecture workshops, pair on complex pipeline work, and build a culture of engineering rigour.
• Partner with MSS to translate domain requirements into technical designs meeting the April 2027 CDMS end-of-support deadline, including historical data migration and downstream consumption patterns.
• Implement data governance using Unity Catalog: access policies, classification, audit logging, and lineage tracking compliant with IM8 and NEA standards.
• Evaluate platform evolution opportunities including serverless compute, Lakebase, and integration with the Weather Data Hub (Azure APIM) for API distribution.

Skills

Databricks

Azure

Python

SQL

Terraform

Spark

Data governance

Leadership

Mentoring

Tools

Unity Catalog

Delta Lake

Structured Streaming

CI/CD

ADLSv2

Job description

GovTech is the lead agency driving Singapore’s Smart Nation initiatives and public sector digital transformation. As the Centre of Excellence for Infocomm Technology and Smart Systems (ICT & SS), GovTech develops the Singapore Government’s capabilities in Data Science & Artificial Intelligence, Application Development, Smart City Technology, Digital Infrastructure, and Cybersecurity.

At GovTech, we offer you a purposeful career to make lives better where we empower our people to master their craft through robust learning and development opportunities all year round.

Play a part in Singapore’s vision to build a Smart Nation and embark on your meaningful journey to build tech for public good. Join us to advance our mission and shape your future with us today!

Learn more about GovTech at tech.gov.sg.

[Job Overview]

The National Environment Agency (NEA) is the leading public organisation responsible for ensuring a clean and sustainable environment for Singapore. Its key roles are to improve and sustain a clean environment, promote sustainability and resource efficiency, maintain high public health standards, provide timely and reliable meteorological information, and encourage a vibrant hawker culture. NEA works closely with its partners and the community to develop and spearhead environmental and public health initiatives and programmes. It is committed to motivating every individual to care for the environment as a way of life, in order to build a liveable and sustainable Singapore for present and future generations.

We are seeking a Senior Data & Platform Engineer to lead the build-out of NEA's Weather Data Lake, a Databricks-based analytics platform replacing the legacy Climate Data Management System (CDMS) and serving as the single source of truth for meteorological and climate data across the Meteorological Service Singapore (MSS).

As a forward-deployed engineer, you will own the MSS data platform architecture on the Government Commercial Cloud (GCC 2.0) Azure stack, design and deliver production data pipelines, and mentor junior engineers. This role sits at the intersection of data engineering, software engineering, and cloud platform architecture.

As a Senior Data & Platform Engineer, you will be working on:

• Design and build the Weather Data Lake on Databricks (Unity Catalog, Delta Lake, Structured Streaming) running on ADLSv2 within GCC 2.0. Make and document platform architecture decisions covering compute, storage tiering, governance, and cost management.
• Architect production data pipelines ingesting from GTS, ADES, AirViro, IWXXM, and WIS 2.0 sources. Build these using Spark (PySpark) on Databricks with CDC patterns and full data lineage.


• Own the data quality framework. Implement QC pipelines serving both the real-time forecasting path and the post-correction analytics path from a common Azure Blob landing zone.


• Own the infrastructure as code layer, defining all platform resources (Databricks workspaces, storage, networking, IAM) in Terraform with modular configurations. Establish staging and production environment separation with appropriate retention policies and CI/CD pipelines.


• Define integration contracts between the Azure Landing Zone (ingestion) and the Weather Data Lake (analytics), ensuring operational systems do not take direct dependencies on the lake.


• Mentor junior data engineers and software engineers. Conduct code reviews, run architecture workshops, pair on complex pipeline work, and build a culture of engineering rigour.


• Partner with MSS to translate domain requirements into technical designs meeting the April 2027 CDMS end-of-support deadline, including historical data migration and downstream consumption patterns.


• Implement data governance using Unity Catalog: access policies, classification, audit logging, and lineage tracking compliant with IM8 and NEA standards.


• Evaluate platform evolution opportunities including serverless compute, Lakebase, and integration with the Weather Data Hub (Azure APIM) for API distribution.

[What we are looking for]

• 8+ years in data engineering, platform engineering, or backend software engineering, with at least 3 years in a senior or lead capacity.


• Strong hands-on experience with Databricks (Delta Lake, Unity Catalog, Structured Streaming, Workflows) and the lakehouse architecture pattern.


• Deep familiarity with Azure — ADLSv2, Blob Storage, Data Factory, and networking fundamentals (VNet, Private Endpoints, ExpressRoute).


• Strong Terraform experience for data platform infrastructure: provisioning Databricks workspaces, storage, networking, and IAM in a modular, version-controlled manner. Comfortable with CI/CD in regulated cloud environments.


• Proficiency in Python and SQL. Solid hands-on Spark (PySpark or Scala) experience building and optimising large-scale data processing on Databricks.


• Demonstrated ability to make platform architecture decisions, write ADRs, and present trade-offs to technical and non-technical stakeholders.


• Track record of mentoring junior engineers and raising the engineering bar across a team.


• Experience with data governance frameworks, data quality tooling, and lineage tracking in production.


• Experience in GCC 2.0 or equivalent regulated cloud environments with strict change control is highly desirable.


• Strong communication skills bridging data platform concepts with domain experts, product managers, and leadership.

[Nice to have]

• Experience with CDC patterns (Debezium, Delta Live Tables, or similar).


• Familiarity with Azure APIM and how data platforms feed downstream API layers.


• Experience with agentic coding tools such as Claude Code for accelerating development workflows.


• Prior experience in scientific or environmental data domains.


• Knowledge of WMO data standards and WIS 2.0.

What we offer you:

GovTech is an equal opportunity employer committed to fostering an inclusive workplace that values diverse voices and perspectives, as we believe that diversity is the foundation to innovation.

Our employee benefits are based on a total rewards approach, offering a holistic and market-competitive suite of perks. These include leave benefits to meet your work-life needs and employee wellness programs.

We champion flexible work arrangements (subject to your job role) and trust that you will manage your own time to deliver your best, wherever you are, and whatever works best for you.

Learn more about life inside GovTech at go.gov.sg/GovTechCareers.

Stay connected with us on social media at go.gov.sg/ConnectWithGovTech