Senior Data Engineer, Belgium, 12+ Yrs Experience

Nishtech; Brussel Hoofdstad; Sur place; EUR 65 000 - 95 000; Plein temps

Résumé du poste

Nishtech BV is seeking an experienced Data Engineer to design, develop and optimize data pipelines and warehousing solutions. You will work with SQL Server, AWS Redshift, PostgreSQL, Spark, and PySpark to ingest, transform and load large datasets for reporting and analytics.

You will implement ETL frameworks, migrate on‑premise databases to cloud environments, and contribute to data modeling with Hive/HBase. Strong Python (Pandas) and BI tool experience are expected.

Qualifications

• 5–7 years of Data Engineering experience.
• Experience with data warehouses, marts using SQL Server, AWS Redshift, PostgreSQL.
• Expert knowledge of MongoDB NoSQL data modeling and admin tasks.
• Experience with Spark, PySpark, Spark SQL for ingestion and transformations.
• ETL development with Spark, Python (Pandas), Airflow.
• Migrations from on‑prem to cloud for SQL Server, PostgreSQL, MongoDB.
• SQL databases administration and development.
• Data visualization reports with Tableau, Power BI, SSRS.

Responsabilités

• Develop and maintain data warehouses and data marts with multiple business logics.
• Implement ETL pipelines using Spark and Python; load standardized data into Hive/HBase.
• Manage and administer PostgreSQL, SQL Server and AWS Redshift clusters.
• Migrate databases from on‑prem to cloud infrastructure.
• Build and optimize Spark jobs for large datasets using PySpark and Spark SQL.
• Create dashboards and reports using Tableau, Power BI, SSRS.

Connaissances

Data Engineering

SQL Server

AWS Redshift

PostgreSQL

Spark PySpark

ETL with Spark

Hive/HBase

Tableau/Power BI

Python (Pandas)

Airflow

Outils

pg_loader

bcp utility

timescaleDB

Description du poste

Description
Language : English

Required Skills

• 5-7 years or more work experience in Data Engineering
• Develop multiple Data Warehouses, Data marts having various business logic with Microsoft
  SQL Server , AWS Redshift ,AWS RDS and PostgreSQL database.
• Implement advanced concepts like OLTP and OLAP using Microsoft SQL Server, PostgreSQL,
  AWS Redshift and AWS RDS.
• Manage and configured SQL databases like PostgreSQL, Microsoft SQL Server and AWS
  Redshift as an Administrator and Developer.
• Implement Spark Parallelize across multiple spark nodes to help ingest and transform huge
  datasets using PySpark.
• Import data from AWS S3 into Spark RDD.
• Make use of Spark SQL to query data frames and Spark Session data.
• Implement ETL framework using Spark with Python and loaded standardized data into Hive
  and HBase tables.
• Develop various complex ETL flows including Data Extraction , Storing Data , Data
  Warehousing and Dimensional & Data Modeling with SSIS and Python (Pandas , Airflow and
  PySpark).
• Make various migrations of databases from on-premises infrastructure to cloud based
  infrastructure for Microsoft SQL Server, PostgreSQL and Mongo DB databases .
• Expert Knowledge on Mongo DB NoSQL Data modelling , tuning , disaster recovery and
  scaling.
• Made use of PL/SQL scripts using shell scripts to clean , tune and load data into databases.
• Made use of modules like pg_loader , bcp utility and timescale DB parallel copy during
  migrations to automate bulk data transfers.
• Have made use of Shell Scripts to clean source files for consumption , scheduling of jobs and
  transfer of files from source to destination.
• Extract , Load and Transform using PySpark , Spark SQL on Spark Clusters hosted on AWS
  and Databricks.
• Optimization of existing Spark Clusters hosted on AWS and Databricks.
• Developed multiple data visualization reports using Tableau , Power BI and SSRS with filters
  and charts.
• Made use of advanced Power BI features like RLS with binning/grouping , outlier grouping to
  create user specific views.
• Make use of M query to enable custom data ingestion as per customer requirement during
  dataset refresh of Power BI

“NISH Tech BV is an Equal Opportunity Employer. Our policy is clear: there shall be no discrimination on the basis of age, disability, sex, race, religion or belief, gender reassignment, marriage/civil partnership, pregnancy/maternity, or sexual orientation.

We are an inclusive organization and actively promote equality of opportunity for all with the right mix of talent, skills and potential. We welcome all applications from a wide range of candidates. Selection for roles will be based on individual merit alone.”