Senior Data Engineer — AI-Driven Pipelines & Lakehouse

Intuition IT Solutions; Dublin; On-site; EUR 90,000 - 120,000; Full time

Job summary

Intuition IT Solutions is seeking an experienced Data Engineer to drive data quality and testing initiatives. The role focuses on building scalable data pipelines using Databricks, PySpark, and Spark within cloud environments.

You will work with data warehouses, lakes, and lakehouses, implementing dimensional models and robust SQL validation. You will contribute to AI-powered products with LLMs and RAG retrieval systems, operating in an agile team.

Qualifications

• 6+ years of experience in Data Engineering, Data Quality Engineering, or Data Testing.
• Hands-on experience with Databricks and PySpark.
• Strong experience with Hadoop ecosystem components such as Hive, HDFS, Spark, and related Apache technologies.
• Advanced SQL expertise for large-scale data validation and analysis.
• Experience with Data Warehouses, Data Lakes, and Lakehouse architectures.
• Understanding of Star Schema, Snowflake Schema, and dimensional modeling.
• Experience with cloud platforms (Azure, AWS, or GCP).
• Proficiency with Java, Spring Boot, RESTful APIs, Databricks, and Spark/PySpark for data pipelines.
• Hands-on experience building AI-powered products using LLMs, RAG/retrieval systems, orchestration frameworks (e.g., LangChain, LlamaIndex), and tool-calling/function-calling patterns.
• Strong grasp of agentic workflow concepts, including planning/reasoning loops, memory management, grounding, evaluation frameworks, safety mechanisms, and human-in-the-loop design.
• 6+ years of software engineering experience building scalable, distributed systems and cloud-native applications in an agile environment.
• Proven ability to lead team-level design, evaluate technical trade-offs, and translate business needs into secure, maintainable solutions.
• Strong collaboration, communication, and problem-solving skills, comfortable working through ambiguity.
• Bachelor's degree in Computer Science, Software Engineering, or a related field, or equivalent practical experience.

Skills

Data engineering

Data quality engineering

Data testing

Databricks

PySpark

SQL

Data warehouses

Data lakes

Lakehouse architectures

Star schema

Snowflake schema

Dimensional modeling

Cloud platforms

Azure

AWS

GCP

Java

Spring Boot

RESTful APIs

LangChain

LlamaIndex

agentic workflows

planning/reasoning loops

memory management

secure, maintainable solutions

team leadership

agile environment

Education

Bachelor's degree in Computer Science, Software Engineering, or related field

Tools

Databricks

Spark/PySpark

Hadoop ecosystem (Hive, HDFS)

Job description

Intuition IT Solutions is seeking an experienced Data Engineer to drive data quality and testing initiatives. The role focuses on building scalable data pipelines using Databricks, PySpark, and Spark within cloud environments.

You will work with data warehouses, lakes, and lakehouses, implementing dimensional models and robust SQL validation. You will contribute to AI-powered products with LLMs and RAG retrieval systems, operating in an agile team.