BAU Lead - Data Engineering

Espire Infolabs (Singapore) Pte Ltd; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

Espire Infolabs Singapore is seeking a seasoned BAU Lead - Data Engineering to steer data operations in a cloud-first environment. You will lead a team, drive production support, and implement scalable data pipelines using AWS, Snowflake, and Spark, while aligning data strategy with business needs.

The role requires hands-on leadership, deep technical expertise in AWS and Snowflake, and a track record of delivering reliable data platforms and improvements in a dynamic, stakeholder-driven setting.

Qualifications

• 8+ years of data engineering experience.
• Bachelor's degree in computer science or STEM.
• 5+ years as a Lead handling support & production issues.
• 2+ years with large scale datasets and data lake/warehouse techs (AWS Redshift, Google BigQuery, Snowflake).
• 3+ years ETL (AWS Glue), S3, RDS, Kinesis, Lambda, Airflow, Step Functions.
• Agile/customer-facing experience.
• Distributed systems and AWS knowledge.
• Data streaming and scalable processing experience.
• SQL, Python, UNIX shell, Spark proficiency.
• RDBMS, data ingestion, data flows, and data integration understanding.
• SDLC/Lean/Agile experience; CI/CD and Git deployments.
• Excellent communication with technical, business, and management teams.

Responsibilities

• Lead day-to-day BAU data operations, production support, incident management, and platform stability.
• Develop tools to improve data flows between internal/external systems and the data lake/warehouse.
• Collaborate with stakeholders to define data structure, availability, scalability, and accessibility.
• Build robust data ingest pipelines to collect, clean, harmonize, merge, and consolidate data sources.
• Document processes and steps for data pipelines and operations.
• Lead migrations of data transformation jobs to Snowflake/Glue-based stack (and related tech).
• Design and implement data management processes including sourcing, integration, and transformation.
• Collaborate with Data Scientists, Architects and DevOps on analytics projects.
• Ensure data security protocols and regulatory compliance.

Skills

Data team leadership

AWS architecture

Snowflake

BAU operations

Airflow/MWAA

SQL

Python/PySpark

Cloud optimization

Data strategy

Data management design

Data security

Data integration

CI/CD & Git

Education

Bachelor's degree in Computer Science or STEM

Tools

AWS

Snowflake

Airflow

MWAA

Spark

Python

SQL

UNIX shell

Job description

Responsibilities

We are looking for a seasoned BAU Lead - Data Engineering to lead our data operations and support function in a cloud-first environment. The ideal candidate will bring strong technical leadership, hands-on expertise in AWS and Snowflake, and proven experience in managing Business-As-Usual (BAU) data platforms, production support, and continuous improvement initiatives.

• Develop tools to improve data flows between internal/external systems and the data lake/warehouse.
• Work with stakeholders to understand needs for data structure, availability, scalability, and accessibility.
• Build robust and reproducible data ingest pipelines to collect, clean, harmonize, merge, and consolidate data sources.
• Understanding existing data applications and infrastructure architecture
• Build and support new data feeds for various Data Management layers and Data Lakes
• Evaluate business needs and requirements.
• Support migration of existing data transformation jobs in Oracle, and MS-SQL to Snowflake.
• Lead the migration of the existing data transformation jobs in Oracle, Hive, Impala etc. into Spark, Python on Glue etc.
• Able to document the processes and steps.
• Develop and maintain datasets.
• Improve data quality and efficiency.
• Lead Business requirements and deliver accordingly.
• Collaborate with Data Scientists, Architect and Team on several Data Analytics projects.
• Collaborate with DevOps Engineer to improve system deployment and monitoring process.
• Experience in critical production support and how the BAU functions is preferred.

Key Skills

• Lead a team of data engineers in managing day-to-day BAU operations, production support, incident management, and platform stability.
• Strong AWS knowledge in terms of designing new architecture and providing optimized solutions for existing ones. (S3, Glue, DMS, MWAA, AMS, IAM).
• In-depth knowledge with respect to Snowflake and its architecture.
• Prefer prior Experience in BAU environment.
• Good knowledge on Airflow and MWAA.
• Hands-on experience in SQL/Python/Pyspark.
• Expertise in optimizing techniques in cloud environments.
• Should have the vision on data strategy and able to deliver the same.
• Should be able to lead the design and implementation of data management processes, including data sourcing, integration, and transformation.
• Able to manage and lead a team of data professionals, providing guidance, mentoring and foster a collaborative and innovative team culture focused on continuous improvement.
• To evaluate and recommend data-related technologies, tools, and platforms.
• Collaborate with IT teams to ensure seamless integration of data solutions.
• Should have experience in Implementing and enforcing data security protocols and ensure compliance with relevant regulations.

Required Qualifications

• At least 8+ years of Data Engineer Experience
• Bachelor qualification in a computer science or STEM (science, technology, engineering, or mathematics) related field.
• At least 5+ years of recent hands-on professional experience (actively coding) working as a Lead handling support & production issue.
• 2+ experience with large scale datasets, data lake and data warehouse technologies such as AWS Redshift, Google BigQuery, Snowflake. Snowflake is highly preferred.
• At least 3+ years of experience in ETL (AWS Glue), Amazon S3, Amazon RDS, Amazon Kinesis, Amazon Lambda, Apache Airflows, Amazon Step Functions.
• Professional experience working in an agile, dynamic and customer-facing environment is required.
• Understanding of distributed systems and cloud technologies (AWS) is highly preferred.
• Understanding of data streaming and scalable data processing is preferred to have.
• Strong knowledge in scripting languages like SQL ,Python, UNIX shell and Spark is required.
• Understanding of RDBMS, Data ingestions, Data flows, Data Integrations etc.
• Technical expertise with data models, data mining and segmentation techniques.
• Experience with full SDLC lifecycle and Lean or Agile development methodologies.
• Knowledge of CI/CD and GIT Deployments.
• Ability to work in team in diverse/ multiple stakeholder environment.
• Ability to communicate complex technology solutions to diverse teams namely, technical, business and management teams

Soft Skills

• Ability to work in a collaborative environment and coach other team members on coding practices, design principles, and implementation patterns that lead to high quality maintainable solutions.
• Excellent communications and stake-holder management are required.
• Ability to handle senior leadership and report to them if required.
• Ability to work in a dynamic, agile environment within a geographically distributed team.
• Ability to focus on promptly addressing customer needs.
• Ability to work within a diverse and inclusive team.
• Technically curious, self-motivated, versatile and solution-oriented