Data Engineer

One North, a Singapore based firm specializing in providing AI based Technology Solutions is currently hiring Data Engineers with about 5+ years of experience as per details given below.

Skills & Responsibilities:

As a Data Engineer, you will focus on managing the Hadoop cluster, implementing data ingestion framework, and designing data models.

Your primary role will be to implement data lake & transform the data for business use.

• Experience with at least one Cloud Infra provider (Azure/AWS)
• Experience in building data pipelines using batch processing with Apache Spark (Spark SQL, DataSet / Dataframe API) or Hive query language (HQL)
• Knowledge of Big data ETL processing tools
• Write SQL scripts to test ETL results
• Write scripts to regression test the ETL iterations already completed
• Experience with Hive and Hadoop file formats (Avro / Parquet / ORC)
• Basic knowledge of scripting (shell / bash)
• Experience with CI/CD tools such as Jenkins, JIRA, Bitbucket, Artifactory, Bamboo, and Azure DevOps.
• Understanding of DevOps practices using Git version control
• Debug, fine-tune and optimize large scale data processing jobs
• Deliver big data solutions based on premise Hadoop or cloud-based systems like AWS.
• Manage Hadoop cluster, participate in scale out planning & implementation.
• Design ingestion layer for structured & unstructured data (text, voice, xml etc)
• Implement specific data model for business & analytics use.
• Deliver ELT solution including data extraction, transformation, cleansing, data integration, and data management.
• Augment with new sources of data including internal/external untapped data.
• Contribute to the establishment and maintenance of cloud computing platform and big data services.
• Provide support for analytics tools & environment like RServer etc & debug performance issues.

Competencies:

• Databases: RDBMS, SQL programming
• ETL and Data Integration Tools: Azure Data Factory (ADF), Microsoft SQL Server Integration Service (SSIS), SAS Data Integration
• Big Data: Hadoop (Hortonworks), Hive, Spark, Sqoop, etc
• Programming and Scripting: Linux/Unix Shell Scripting, Java, Scala, Hive QL
• BI/Dashboarding: SAS Visual Analytics, Qliksense

Working Experience:

• 5+ years (no upper limit) in data engineering and modelling.
• Hands-on in MS SQL. Should be an expert in MS SQL queries and scripting.
• Hands-on experience in testing ETL systems is mandatory
• Hands-on in managing data mapping, data quality and integrity, performance in data processing.
• Develop and implement techniques and analytics applications to transform raw data into meaningful information
• Experience in Agile software development.
• Candidate should be able to create executable scripts for regression testing purposes
• Experience in testing SSIS ETL mappings is good to have
• Visualize, interpret, and report data findings and may create dynamic data reports as well.