Senior Data Engineer

Creditsafe Group; Dublin; Hybrid; EUR 90,000 - 120,000; Full time

Benefits offered by this job

Bonus Scheme

Cycle to work and Wellbeing Programme

Global company gatherings and events

E-learning and career progression

Job summary

Creditsafe is expanding its Consumer Bureau with a Data Engineer who will join a team of highly skilled engineers building Creditsafe’s new data platform. You will work on data quality, lineage, and auditability, using AWS technologies like S3, Iceberg, Aurora, DynamoDB, Lambda, Fargate, ECS, Airflow, Spark, and Python-based processing pipelines.

You will help deliver fast, scalable APIs for clients. The role emphasizes end-to-end ownership, production-grade data pipelines, and mentoring

Qualifications

• Experience designing metadata-driven, event-based data platforms.

Responsibilities

• Actively contribute to the codebase and peer reviews.
• Design and build metadata driven, event based distributed data processing platform using Python, Spark, Airflow, Iceberg, DynamoDB, AWS Glue, S3.
• Build and scale data pipelines for sub-100ms latency serving.
• Understand domain data to improve product.
• Leverage AI tools to aid in work.

Skills

Data engineering

Python programming

Communication skills

Tools

Python

Spark

Airflow

Iceberg

DynamoDB

AWS Glue

S3

Athena

Redshift

Lambda

API Gateway

Terraform

CI/CD

Job description

We are looking to expand our Consumer Bureau team, are you an experienced Data Engineer?

About Creditsafe

With the largest owned database supported by a partner network, Creditsafe provides data on 320 million companies in over 100 countries to support its customers in this regard. Creditsafe is scaling business and currently going through a significant growth stage. Being a data company, Creditsafe collects and processes large amounts of complex data, received from multiple channels with the only purpose of serving our clients with opportunities to gain actionable insights. Our holistic data aggregation helps our clients to fill the knowledge gaps, influence decision making, and find hidden patterns.

About Opportunity

This opportunity is to join a team of highly technically skilled engineers who are designing Creditsafe’s new data platform with data quality, data lineage, and auditability as the primary goals whilst maintaining high throughput and scalability. The data platform will be built using AWS infrastructure such as S3, Iceberg, Aurora, DynamoDB, Lambda, Fargate, ECS, Airflow, Spark. The platform is expected to manage over one hundred million objects with a high volume of daily updates handling addition, deletion, and correction of our data and indexes in an auditable manner. The platform will employ a sophisticated matching algorithm to enable us to assign incoming updates to existing objects. Our data processing application is entirely based on Python and designed to efficiently transform incoming raw data volume into API consumable schema. The team is also building highly available and low latency APIs to enable our clients with faster data delivery.

What you will be doing?

• You will actively contribute to the codebase and participate in peer reviews.
• Design and build metadata driven, event based distributed data processing platform using technologies such as Python, Spark, Airflow, Iceberg, DynamoDB, AWS Glue, S3.As an experienced Engineer, you will play a critical role in the design, development, and deployment of our business-critical system.
• You will be building and scaling Creditsafe’s data pipelines combining batch and streaming approaches and targeting sub-100ms latency serving.
• Understanding domain data to make recommendations to improve the existing product.
• Employing various AI tools such as Perplexity, Cursor, CoPilot, etc to aid in all aspects of your work

What are we looking for?

• You understand and can implement data engineering best practices.
• You demonstrate the ability to write clean & efficient code and combine it with the cloud environment for best performance.
• Proven experience of development experience within a commercial environment creating production grade data pipelines in python.
• We appreciate someone that is looking to grow your skills through daily technical challenges and enjoy problem solving, whiteboarding in collaboration with team.
• You have excellent communication skills, and ability to explain your views clearly to the team and are open to understanding the views of others.
• You have a proven track record to draw from a deep and broad technical expertise to mentor engineers, complete hands‑on technical work, and provide leadership on complex technology issues.
• You share your ideas collaboratively via wikis, discussions boards, etc and share any decisions made, for the benefit of others.
• Be able to take ownership of end-to-end deliverables from planning all the way to production.

Why Creditsafe?

• Career Progression: Clearly defined career progression path with opportunities to grow as a technical leader or move into adjacent roles.
• Learning & Development: Access to a plethora of learning resources and courses to perpetually enhance your skills, with weekly dedicated training and focus time.
• Work-Life Balance: Committed to promoting a balanced lifestyle with flexible working hours and hybrid work options.
• Innovation: Engage in exciting projects, harnessing the power of the latest technologies in cloud computing and data management.

Our Technology Stack: Python, Linux, Airflow, AWS DynamoDB, S3, Glue, Athena, Redshift, lambda, API Gateway, Terraform, CI/CD

• Bonus Scheme
• Cycle to work and Wellbeing Programme
• Global Company gatherings and events
• E-learning and excellent career progression opportunities
• Plus more that can be found on the benefits section on the Careers page, https://creditsafecareers.kinsta.cloud/gb/