Data Engineer (Porto ou Lisboa, 3y-6y)

act digital; Lisboa; Presencial; EUR 70 000 - 100 000; Tempo integral

Resumo da oferta

act digital is seeking a Senior GCP Data Engineer to drive a strategic data transformation and cloud migration, migrating from SQL Server to Google Cloud Platform. You will design scalable pipelines with BigQuery, Python, and Cloud Composer, while collaborating with DBAs, analysts, and business stakeholders to rebuild data flows for a modern cloud data platform.

You will architect end-to-end data migration, ensure data quality and performance, and help establish engineering standards for the new

Qualificações

• StrongData Engineering experience required.
• Advanced knowledge of Google Cloud Platform and its data services.
• Significant hands-on BigQuery experience including data modeling.
• Complex SQL development and performance tuning.
• Partitioning and clustering for large datasets.
• Cost management and optimization in cloud data warehousing.
• Hands-on experience with Google Cloud Composer for DAGs.

Responsabilidades

• Migrate data and ETL processes from SQL Server to Google Cloud Platform.
• Design and build scalable data pipelines on cloud technologies.
• Develop optimized SQL and data models for BigQuery.
• Automate migration, transformation, validation and integration using Python.
• Ensure data quality, integrity, performance and reliability during migration.
• Establish engineering standards and best practices for the new GCP data platform.
• Support teams and users during the cloud transition.
• Collaborate with DBAs, data analysts, scientists and business stakeholders.

Conhecimentos

Data Engineering

Google Cloud Platform

BigQuery

SQL

Python

ETL/ELT

Data Migration

Data Integration

Data Modeling

Data Quality

Dataflow

Ferramentas

Google Cloud Composer

BigQuery

GCP APIs

Dataflow

Descrição da oferta de emprego

We are looking for a Senior GCP Data Engineer to support a strategic data transformation and cloud migration initiative.

In this role, you will help migrate databases, ETL processes, and analytical workloads from SQL Server to Google Cloud Platform, designing a modern, scalable, and cost-efficient data ecosystem powered by BigQuery, Cloud Composer, and Python.

You will work closely with Database Administrators, Data Engineers, Analysts, Data Scientists, and business stakeholders to understand the existing environment, rebuild critical data flows, and ensure a smooth transition to the new cloud platform.

This is an excellent opportunity for someone who combines strong data engineering expertise with hands-on cloud migration experience and enjoys solving complex data, performance, and integration challenges.

As a Senior GCP Data Engineer, you will:

• Contribute to the migration of data and ETL processes from SQL Server to GCP.
• Design and build scalable data pipelines using modern cloud technologies.
• Develop optimized data models and queries for BigQuery.
• Automate migration, transformation, validation, and integration activities using Python.
• Guarantee data quality, integrity, performance, and operational reliability throughout the migration.
• Help establish engineering standards and best practices for the new GCP data platform.
• Support teams and end users during the transition to the cloud environment.

Data & Database Migration

• Contribute to the design and execution of the migration strategy from SQL Server to Google Cloud Platform.
• Analyze existing databases, ETL workflows, stored procedures, dependencies, and data structures.
• Develop migration scripts and pipelines to transfer and transform data into BigQuery.
• Rebuild or adapt existing SQL Server processes for a cloud-native data architecture.
• Support migration planning, technical assessments, testing, reconciliation, and production deployment.
• Ensure data consistency and integrity before, during, and after migration.
• Identify migration risks, technical dependencies, and potential performance issues.
• Design, develop, and maintain robust ETL and ELT pipelines on Google Cloud Platform.
• Replace or modernize existing data flows using scalable cloud services.
• Build reusable components and standardized pipeline patterns.
• Ensure pipelines are reliable, maintainable, observable, and performance-oriented.
• Implement appropriate retry, recovery, logging, and error-handling mechanisms.
• Support both batch processing and data integration use cases.

Workflow Orchestration

• Create, schedule, monitor, and maintain workflows using Google Cloud Composer.
• Design and manage DAGs for migration, transformation, and integration processes.
• Define dependencies, execution schedules, retries, alerts, and failure-handling processes.
• Troubleshoot orchestration issues and optimize workflow execution.
• Ensure reliable operationalization of pipelines across development, testing, and production environments.

BigQuery Engineering & Optimization

• Design and implement scalable data structures in BigQuery.
• Develop and optimize complex SQL queries for analytical and transformation workloads.
• Adapt T-SQL logic and existing SQL Server processes to BigQuery SQL.
• Apply BigQuery optimization techniques, including:Partitioning
• Query optimization
• Storage optimization
• Monitor query performance and resource consumption.
• Contribute to the design of cloud data models aligned with business and analytical requirements.

Python Development & Automation

• Develop Python scripts for:Data migration
• Data transformation
• Integration with GCP services and APIs
• Build reusable and maintainable Python components.
• Automate repetitive migration and operational activities.
• Support the development of validation and monitoring utilities.
• Apply software engineering best practices, including code reviews, version control, testing, and documentation.

Data Quality & Validation

• Implement data quality controls throughout the migration lifecycle.
• Develop automated reconciliation processes between source and target systems.
• Validate data completeness, accuracy, consistency, and integrity.
• Define tests for migrated tables, transformations, pipelines, and business rules.
• Investigate data discrepancies and coordinate their resolution.
• Ensure that the new platform produces reliable and trusted data for its users.

Monitoring, Support & Troubleshooting

• Monitor data pipelines, scheduled workflows, and production processes.
• Investigate and resolve pipeline failures, data issues, and performance bottlenecks.
• Support debugging and incident resolution during the migration and after production deployment.
• Perform root cause analysis and implement sustainable corrective measures.
• Improve platform observability, alerts, operational procedures, and support documentation.

Collaboration & Stakeholder Support

• Work closely with SQL Server administrators and existing technical teams to understand the current ecosystem.
• Collaborate with Data Analysts, Data Scientists, and business teams to gather data requirements.
• Ensure that the new GCP platform meets functional, analytical, and operational expectations.
• Participate in technical design sessions, code reviews, and testing activities.
• Communicate migration progress, risks, dependencies, and technical decisions clearly.
• Support knowledge transfer and the adoption of the new cloud data platform.
• Stay informed about GCP data engineering capabilities and cloud migration practices.
• Identify opportunities to improve performance, scalability, cost efficiency, and maintainability.
• Propose new tools, patterns, and automation opportunities for the data platform.
• Contribute to engineering standards, documentation, and reusable development practices.
• Help establish a culture of technical excellence and continuous improvement.

What We’re Looking For

Mandatory Requirements

• Strong professional experience in Data Engineering.
• Advanced knowledge of Google Cloud Platform, particularly its data services.
• Significant hands-on experience with BigQuery, including:Data modeling
• Complex SQL development
• Partitioning and clustering
• Query performance optimization
• Cost management
• Strong experience with Google Cloud Composer.
• Hands-on experience creating, scheduling, monitoring, and debugging DAGs.
• Excellent SQL skills and the ability to adapt existing SQL workloads to BigQuery.
• Advanced Python skills for data engineering, automation, and GCP integration.
• Experience designing and building ETL or ELT pipelines.
• Strong understanding of database and data migration methodologies.
• Ability to analyze legacy or on-premises environments and redesign them for the cloud.
• Strong analytical and troubleshooting capabilities.
• Good English communication skills, with a minimum B2 level.
• BigQuery
• Dataflow
• GCP APIs
• Database Migration Service
• ETL
• ELT
• Batch Data Processing
• Data Integration
• Data Migration

Development

• Python
• SQL
• T-SQL
• Automation Scripting
• API Integration

Databases & Data Warehousing

• Microsoft SQL Server
• BigQuery
• Relational Databases
• Cloud Data Warehouses
• Data Modeling
• Data Vault

Performance & Cost Optimization

• BigQuery Partitioning
• BigQuery Clustering
• Query Optimization
• Storage Optimization

Quality & Delivery

• Data Quality
• Data Validation
• Automated Testing
• Version Control
• CI/CD Practices
• Technical Documentation

Nice to Have

• Direct experience with Microsoft SQL Server, including:SQL Server architecture
• T-SQL
• Stored procedures
• Basic database administration
• Experience with additional GCP services such as:Cloud Storage
• Dataflow
• Database Migration Service
• Experience using cloud-specific database migration tools.
• Knowledge of Data Warehousing principles and dimensional modeling.
• Experience with Data Vault methodology.
• Knowledge of Power BI or other data visualization tools.
• Experience with source control and collaborative development workflows.
• Familiarity with CI/CD pipelines for data engineering solutions.
• Previous involvement in large-scale on-premises to cloud migration programs.