AI-Driven Data Engineer & Lakehouse Lead

Fulcrum Digital Inc

Dublin

On-site

EUR 90,000 - 130,000

Full time

3 days ago

Be an early applicant

Application generator

Turn this role into an interview — a resume and cover letter built around what this employer wants.

Get past ATS filters

Job summary

Fulcrum Digital Inc. is seeking a Senior Data Engineer to design, build, and scale data foundations powering AI-powered services.

You will blend hands-on data engineering with a focus on data quality, spanning Databricks, PySpark, Hadoop, and Lakehouse architectures to ensure accurate, reliable data platforms. You will lead technical design, mentor engineers, and ship data-driven products end-to-end, collaborating with cross-functional teams and driving AI-enabled capabilities across LLM

Qualifications

Bachelor's degree in Computer Science, Software Engineering, or a related field, or equivalent practical experience.
6+ years of experience in Data Engineering, Data Quality Engineering, or Data Testing.
Hands‑on experience with Databricks and PySpark.
Strong experience with Hadoop ecosystem components such as Hive, HDFS, Spark, and related Apache technologies.
Advanced SQL expertise for large-scale data validation and analysis.
Experience with Data Warehouses, Data Lakes, and Lakehouse architectures.
Proficiency with Java, Spring Boot, RESTful APIs, Databricks, and Spark/PySpark for data pipelines.
Hands‑on experience building AI‑powered products using LLMs, RAG/retrieval systems, orchestration frameworks (e.g., LangChain, LlamaIndex), and tool‑calling/function‑calling patterns.

Responsibilities

Perform end-to-end validation of data pipelines across ingestion, transformation, and consumption layers.
Execute source-to-target reconciliation and data quality checks, identifying and resolving anomalies.
Define and implement data quality frameworks, metrics, and controls.
Develop and validate data pipelines using PySpark and Databricks across Hadoop, Hive, and Lakehouse environments.
Support ETL/ELT workflows and optimise data processing jobs for performance and scalability.
Write advanced SQL queries for data profiling, reconciliation, and root cause analysis, including complex joins, window functions, CTEs, and aggregations.
Validate and monitor data across Databricks Lakehouse architecture on cloud platforms such as Azure, AWS, or GCP.
Analyze production issues, conduct root cause analysis, and implement proactive monitoring and automated quality checks.
Lead team-level design for AI-powered services, including LLM and agent architecture, RAG pipelines, tool-calling integrations, and data flows.
Design, build, and productionise AI-powered features, including LLM integrations, RAG/retrieval systems, and agentic workflows using frameworks such as LangChain and LlamaIndex.
Implement agentic workflow patterns, including planning/reasoning loops, memory management, grounding, evaluation frameworks, safety mechanisms, and human-in-the-loop design.
Serve as a hands‑on technical contributor across the full engineering lifecycle, from design and coding to testing, debugging, and delivery.
Identify and resolve performance bottlenecks across services, APIs, data pipelines, and AI/LLM workloads.
Collaborate with cross‑functional teams to translate business requirements into scalable, AI-driven technical solutions.
Mentor junior and mid‑level engineers through technical guidance, pairing, code reviews, and knowledge sharing.
Improve engineering quality through code reviews, design reviews, automated testing, secure coding practices, and observability.