Databricks Team Lead: Data Platform & Engineering

Citco Group of Companies; Dublin; Hybrid; EUR 110,000 - 150,000; Full time

Job summary

Citco Group of Companies is seeking a Team Lead - Data Engineer to direct a skilled team delivering enterprise Databricks solutions on a cloud-based Lakehouse platform. You will guide architecture, data ingestion, streaming pipelines, and security governance while collaborating across global locations.

The role requires hands-on Databricks experience, strong Python/Spark skills, and a proven track record in leading data engineering teams within AWS cloud environments.

Qualifications

• 8+ years of experience in data engineering.
• 3+ years hands-on experience with the Databricks platform.
• Proven experience leading a team.
• Strong expertise in Python and Spark programming.
• Demonstrable experience in using AI in development.
• Proven experience with AWS or other cloud services.
• Deep understanding of data modeling and SQL.
• Experience with Delta Lake and Lakehouse architecture.
• Strong knowledge of ETL/ELT principles and patterns.
• Experience with Git version control.
• Demonstrated ability to optimize data pipelines.
• Excellent communication and collaboration abilities.

Responsibilities

• Lead and mentor a team of data engineers, conducting code reviews and knowledge-sharing sessions across locations.
• Drive the Agile/Scrum SDLC process and collaborate with team members.
• Design and develop Databricks solutions leveraging Lakehouse architecture for enterprise data processing and analytics.
• Develop and optimize ETL/ELT pipelines.
• Create and manage structured streaming pipelines for real-time data processing.
• Configure and optimize Databricks clusters and Spark jobs for performance.
• Utilize Delta Live Tables for data ingestion and transformations.
• Apply Unity Catalog features and IAM best practices for security governance and access control.
• Support infrastructure and resource management using Terraform.
• Implement monitoring solutions for pipeline performance and data quality.
• Contribute to code reviews and knowledge-sharing sessions.

Job description

Citco Group of Companies is seeking a Team Lead - Data Engineer to direct a skilled team delivering enterprise Databricks solutions on a cloud-based Lakehouse platform. You will guide architecture, data ingestion, streaming pipelines, and security governance while collaborating across global locations.

The role requires hands-on Databricks experience, strong Python/Spark skills, and a proven track record in leading data engineering teams within AWS cloud environments.