Cloud Data Engineer - Spark, Lakehouse & AI Workloads

Betaflow Consulting; Lisboa; On-site; EUR 50,000 - 75,000; Full time

Job summary

Betaflow Consulting seeks a skilled data engineer to design and implement cloud-native data solutions that empower analytics, reporting, and AI workloads in Lisbon. You will drive scalable data pipelines on AWS and Databricks, shape modern Lakehouse architectures, and collaborate with cross-functional teams to deliver robust data platforms.

The role emphasizes building reliable, scalable systems, applying best practices for security and governance, and enabling ML/AI workloads through automated

Responsibilities

• Design, develop, and support large-scale cloud-native data solutions on AWS and Databricks.
• Build and maintain batch, streaming, and real-time data pipelines supporting analytics, reporting, machine learning, and AI workloads.
• Engineer scalable distributed data processing systems using Apache Spark and related Big Data technologies.
• Design and implement modern Lakehouse architectures leveraging Databricks, Delta Lake, and Apache Iceberg.
• Lead modernization initiatives migrating data workloads from Hadoop-based platforms to cloud-native architectures.
• Develop reusable frameworks and engineering patterns that improve scalability, reliability, and operational efficiency.
• Partner with Architects, Product Managers, Data Scientists, and Software Engineers to define and implement cloud data solutions.
• Evaluate technologies and approaches for data ingestion, transformation, storage, governance, and consumption.
• Implement best practices around security, governance, performance optimization, observability, and cost management.
• Drive automation using Infrastructure-as-Code, CI/CD pipelines, and modern software engineering practices.
• Contribute to the evolution of Marketing Services' cloud and data strategy while enabling future AI and machine learning capabilities.

Job description

Betaflow Consulting seeks a skilled data engineer to design and implement cloud-native data solutions that empower analytics, reporting, and AI workloads in Lisbon. You will drive scalable data pipelines on AWS and Databricks, shape modern Lakehouse architectures, and collaborate with cross-functional teams to deliver robust data platforms.

The role emphasizes building reliable, scalable systems, applying best practices for security and governance, and enabling ML/AI workloads through automated