Data Engineer GCP

Decskill; Porto; Presencial; EUR 60 000 - 90 000; Tempo integral

Resumo da oferta

Decskill is seeking a Data Engineer specializing in Google Cloud Platform to lead the migration of data and ETL processes from SQL Server to GCP. You will design, build, and optimize pipelines on BigQuery, orchestrated via Cloud Composer, ensuring data quality and scalable architectures.

You will work closely with SQL Server admins, analysts, and data scientists to deliver robust data platforms, implement automation with Python, and model data for cloud-native analytics.

Qualificações

• Significant experience with Google Cloud Platform (GCP) including BigQuery design and performance optimization.
• Solid experience in creating, scheduling, monitoring, and debugging DAGs to orchestrate data flows.
• Strong SQL skills; ability to optimize queries for BigQuery and migrate from SQL Server.
• Proficient in Python for migration automation and data transformation tasks.

Responsabilidades

• Participate in design and execution of data and ETL migration from SQL Server to GCP.
• Develop and optimize pipelines for transferring and transforming data into BigQuery.
• Design and maintain robust ETL/ELT pipelines on GCP and orchestrate with Cloud Composer.
• Develop and optimize SQL queries and data schemas for BigQuery during migration.
• Use Python to automate migration tasks and ensure data quality during migration.
• Collaborate with SQL Server admins, analysts, and data scientists to meet user needs.
• Participate in code reviews, best practices, and testing strategies for the platform.
• Ensure monitoring, debugging, and incident resolution for data pipelines during transition and production.

Conhecimentos

BigQuery

Cloud Composer

SQL

Python

Data Migration

Analytical Skills

English (B2)

GCP

Spark

Java

Scala

Ferramentas

Power BI

SQL Server

Git

Dataflow

Database Migration Service

Descrição da oferta de emprego

Decskill, founded in 2014 as an IT Consulting Company, places paramount importance on its greatest asset: its people. Our main mission is to deliver value through knowledge and talent, and we achieve this by fostering a culture of excellence and investing in the development and well-being of our people. With over 600 dedicated professionals and offices in Lisbon, Porto, Madrid, and Luxembourg, Decskill operates across three core areas:

Decskill Talent: We believe that our people are key to our success.

Through Decskill Talent, we empower our team to embrace the digital transformation challenges of our clients. We collaborate with clients to drive innovation, ensuring project success and business growth.

Decskill Boost: Equipping our team with the latest tools and methodologies, we optimize Time-to-Market and deliver innovative solutions exceeding client expectations.

Decskill Connect: Our team collaborates closely with clients to implement and manage IT infrastructures that generate long‑term value.

At Decskill, we believe that by nurturing and empowering our people to confront the challenges of digital transformation, we create value not only for our clients but also for our entire ecosystem, fostering a digital community dedicated to growth and progress.

We are looking for a Data Engineer GCP

Responsibilities:

• Participate in the design and execution of the data and ETL process migration strategy from SQL Server to GCP;
• Develop and optimize scripts and pipelines for efficiently transferring and transforming data into BigQuery;
• Design and Development of Data Pipelines on GCP:
  • Design, develop, and maintain robust and high‑performing ETL/ELT pipelines on GCP, replacing or rebuilding existing flows;
  • Orchestrate and schedule these pipelines using Google Cloud Composer to ensure reliable migration and operationalization.
• Database Management and Optimization on GCP:
  • Develop and optimize complex SQL queries for BigQuery, considering the specificities of migrating from SQL Server to BigQuery (e.g., optimizing partitioning, clustering);
  • Contribute to the design and implementation of optimized data schemas for BigQuery, adapted to new cloud architectures.
• Development and Automation:
  • Use Python to develop scripts for automating migration tasks, data transformation, and integration with other GCP services;
  • Implement processes to ensure data quality and integrity during and after migration.
• Collaboration and Support:
  • Collaborate closely with current teams (SQL Server administrators, analysts) to understand the existing environment and facilitate the transition;
  • Exchange with end-users (business lines, data scientists) to understand their needs and ensure the new platform meets their expectations;
  • Participate in code reviews, development best practices, and testing strategies for the new platform;
  • Ensure monitoring, debugging, and incident resolution related to data pipelines during the transition phase and upon production deployment.
• Technological Watch and Optimization:
  • Stay informed about best practices for migration and the latest developments on Google Cloud Platform;
  • Propose improvements and optimizations for the new Data platform on GCP.

Requirements:

Mastery of Google Cloud Platform (GCP):

• Big Query: Significant experience in schema design, writing complex queries (SQL), performance optimization, and cost management in a cloud context;
• Cloud Composer: Solid experience in creating, scheduling, monitoring, and debugging DAGs to orchestrate data flows, particularly for migration and integration processes;
• SQL: Excellent mastery of the SQL language. Experience with SQL Server is a plus, but more importantly, the ability to adapt and optimize SQL for Big Query;
• Python: In-depth mastery of the Python language for developing scripts (migration, automation, data transformation) and interacting with GCP APIs;
• Data Migration: Experience or strong understanding of the challenges and methodologies for migrating databases from an on‑premise environment (SQL Server) to a cloud environment (GCP);
• Analytical and Problem‑Solving Skills: Ability to analyze existing systems (SQL Server), understand business needs, design solutions on GCP, and resolve complex problems related to migration and pipeline operation;
• Languages (Spark, R, Python, Java, Scala);
• Hadoop tools manipulation, Data engineering process implementation using NoSQL / Sql, Databases and Hadoop tools;
• Good communication skills in English (B2 level required).

Additional Skills (Good to Have):

• Direct knowledge of SQL Server (architecture, T‑SQL, basic administration);
• Experience with other GCP services relevant to data migration and integration (Storage, Dataflow, etc.);
• Knowledge of GCP‑specific migration tools and strategies (e.g., Database Migration Service);
• Experience with data visualization tools (Power BI);
• Understanding Data Warehousing concepts and data modeling (Data Vault);
• Experience with code versioning.

Decskill is committed to equality and non‑discrimination with all our talents. We recruit and promote talent, based on diversity and inclusion, regardless of age, gender, ethnicity, race, nationality or any other form of discrimination incompatible with the dignity of the human being.