Senior Data Engineer: Spark, Databricks & AWS Pipelines

EXASOFT CONSULTING PTE. LTD.

Singapore

On-site

SGD 90,000 - 170,000

Full time

3 days ago

Be an early applicant

Application generator

Don’t send a generic resume — generate a resume and cover letter tailored to this exact role.

Get past ATS filters

Job summary

EXASOFT CONSULTING PTE. LTD. seeks an experienced Data Engineer to design, build, and optimize scalable data pipelines for ingesting, transforming, and delivering high-quality data across cloud and on-premises platforms.

You will work with Databricks, PySpark, and Apache Spark to process large volumes of data, create reliable ETL/ELT workflows, and implement Delta Lake-based data layers. Proficiency in SQL and Python is required.

Qualifications

5+ years of experience in data engineering, ETL development, data integration or related field.
Strong hands-on experience with Databricks and data engineering workloads.
Hands-on experience in PySpark and Apache Spark for distributed data processing.
Solid experience developing ETL/ELT pipelines and data engineering solutions.
Strong experience SQL skills, including complex queries, joins, aggregations, optimization, and data transformation.
Hands-on experience with AWS cloud services used for data engineering and data processing.
Experience with Delta Lake, Delta Live Tables (DLT) is preferred.
Hands-on experience with IBM DataStage and Informatica PowerCenter.
Experience working with relational databases such as Oracle and IBM DB2.
Python for data processing, automation, or pipeline development.
Experience with data modelling, data warehousing, data quality, and data integration concepts.
Hands-on experience in Lambda, Redshift, IAM, CloudWatch, Glue, EC2.
Experience with production scheduling, pipeline monitoring, troubleshooting, and deployment processes.
Experience in version control and CI/CD practices such as TWS, GIT, Jenkins etc.
Ability to work effectively in an Agile/SDLC environment and collaborate with cross-functional teams.
Data brick certification would be preferred.

Responsibilities

Design, develop, and maintain scalable data pipelines for ingesting, transforming, validating, and delivering structured and semi-structured data.
Develop data engineering solutions using Databricks, PySpark, and Apache Spark for large-volume data processing and transformation.
Build and manage ETL/ELT workflows using modern data engineering platforms as well as enterprise ETL technologies.
Develop data transformation and processing logic using SQL and Python, with a focus on performance, reliability, and data accuracy.
Work with Delta Lake and Delta Live Tables to develop and maintain reliable data pipelines and curated data layers.
Develop and optimize ETL workflows using IBM DataStage and Informatica PowerCenter where required.
Work with relational databases including Oracle and IBM DB2 for data extraction, transformation, loading, querying, and performance optimization.
Implement data integration solutions across cloud and enterprise data platforms.
Develop and support data pipelines and associated services on AWS, including data storage, processing, monitoring, and related cloud services.
Perform data validation, reconciliation, quality checks, and troubleshooting to ensure the accuracy and completeness of datasets.
Optimize Spark jobs, SQL queries, ETL workflows, and data pipelines to improve processing efficiency and overall performance.
Collaborate with technical and business teams to understand data requirements and translate them into scalable data solutions.
Follow established development, deployment, documentation, data governance, and SDLC practices.
Monitor production pipelines, investigate failures, and support timely resolution of data processing issues.