Senior Data Engineer - AI-Driven Data Platforms

Fulcrum Digital Inc; Dublin; On-site; EUR 90,000 - 130,000; Full time

Job summary

Fulcrum Digital is seeking a Senior Data Engineer to build and scale the data foundations behind our AI-powered services. You will work across Databricks, PySpark, Hadoop, and Lakehouse architectures to ensure data accuracy, reliability, and performance, shaping AI-driven capabilities and scalable data platforms.

You will lead design, mentor engineers, and ship data-driven products end to end, collaborating with cross‑functional teams and ensuring robust non-functional requirements such as

Qualifications

• 6+ years of experience in Data Engineering, Data Quality Engineering, or Data Testing.
• Hands-on experience with Databricks and PySpark.
• Strong experience with Hadoop ecosystem components such as Hive, HDFS, Spark, and related Apache technologies.
• Advanced SQL expertise for large-scale data validation and analysis.
• Experience working with Data Warehouses, Data Lakes, and Lakehouse architectures.
• Understanding of Star Schema, Snowflake Schema, and dimensional modeling.
• Experience with cloud platforms (Azure, AWS, or GCP).
• Proficiency with Java, Spring Boot, RESTful APIs, Databricks, and Spark/PySpark for data pipelines.
• Hands-on experience building AI-powered products using LLMs, RAG/retrieval systems, orchestration frameworks (e.g., LangChain, LlamaIndex), and tool-calling patterns.
• Strong grasp of agentic workflow concepts, including planning/reasoning loops, memory management, grounding, evaluation frameworks, safety mechanisms, and human-in-the-loop design.
• 6+ years of software engineering experience building scalable, distributed systems and cloud-native applications in an agile environment.
• Proven ability to lead team-level design, evaluate technical trade-offs, and translate business needs into secure, maintainable solutions.
• Strong collaboration, communication, and problem-solving skills, comfortable working through ambiguity.

Responsibilities

• Perform end-to-end validation of data pipelines across ingestion, transformation, and consumption layers.
• Execute source-to-target reconciliation and data quality checks, identifying and resolving anomalies.
• Define and implement data quality frameworks, metrics, and controls.
• Develop and validate data pipelines using PySpark and Databricks across Hadoop, Hive, and Lakehouse environments.
• Support ETL/ELT workflows and optimise data processing jobs for performance and scalability.
• Write advanced SQL queries for data profiling, reconciliation, and root cause analysis, including complex joins, window functions, CTEs, and aggregations.
• Validate and monitor data across Databricks Lakehouse architecture on cloud platforms such as Azure, AWS, or GCP.
• Analyze production issues, conduct root cause analysis, and implement proactive monitoring and automated quality checks.
• Lead team-level design for AI-powered services, including LLM and agent architecture, RAG pipelines, tool‑calling integrations, and data flows.
• Design, build, and productionise AI-powered features, including LLM integrations, RAG/retrieval systems, and agentic workflows using frameworks such as LangChain and LlamaIndex.
• Implement agentic workflow patterns, including planning/reasoning loops, memory management, grounding, evaluation frameworks, safety mechanisms, and human‑in‑the‑loop design.
• Serve as a hands‑on technical contributor across the full engineering lifecycle, from design and coding to testing, debugging, and delivery.
• Identify and resolve performance bottlenecks across services, APIs, data pipelines, and AI/LLM workloads.
• Collaborate with cross‑functional teams to translate business requirements into scalable, AI‑driven technical solutions.
• Mentor junior and mid‑level engineers through technical guidance, pairing, code reviews, and knowledge sharing.
• Improve engineering quality through code reviews, design reviews, automated testing, secure coding practices, and observability.

Skills

Databricks

PySpark

SQL

Data quality

Data testing

Cloud platforms

Java

Spring Boot

LLM integration

AI-powered features

Education

Bachelor's degree in Computer Science or related

Tools

Hive

HDFS

Spark

LangChain

LlamaIndex

Delta Lake

Unity Catalog

Airflow

Kafka

Docker

Kubernetes

Job description

Fulcrum Digital is seeking a Senior Data Engineer to build and scale the data foundations behind our AI-powered services. You will work across Databricks, PySpark, Hadoop, and Lakehouse architectures to ensure data accuracy, reliability, and performance, shaping AI-driven capabilities and scalable data platforms.

You will lead design, mentor engineers, and ship data-driven products end to end, collaborating with cross‑functional teams and ensuring robust non-functional requirements such as