Software Engineer - Data

Needl; Dublin; On-site; EUR 70,000 - 100,000; Full time

Job summary

Needl in Dublin is seeking a Software Engineer to join the Data Engineering team. The role focuses on building robust data processing systems with Python, designing scalable pipelines and handling large datasets in a production environment.

You will work with cloud-based data technologies, relational databases and distributed processing frameworks, contributing to reliable, high-performance data infrastructure.

Qualifications

• 3-5+ years of professional software engineering experience building and maintaining production systems.
• Strong hands-on Python development in production-level codebases.
• Experience with at least one additional language such as Java, C# or Scala.
• Strong SQL skills and solid Relational DB knowledge.
• Experience designing, building or maintaining batch data processing pipelines.
• Experience troubleshooting and supporting production applications.
• Interest in building reliable, scalable data infrastructure.

Responsibilities

• Design, develop and maintain scalable data pipelines and data processing applications.
• Build robust, production-quality software primarily using Python.
• Work with large datasets, relational databases and distributed processing systems.
• Develop and optimise batch processing workflows, ensuring reliability, performance and data integrity.
• Improve the efficiency, scalability and maintainability of existing data infrastructure.
• Work with modern cloud-based data technologies and distributed computing frameworks.
• Collaborate with engineers and stakeholders to deliver high-quality solutions.
• Take ownership of code throughout the SDLC, including testing and deployment.

Skills

Python development

SQL

Production software

Batch data processing

Problem solving

Distributed data technologies

Tools

Databricks

Delta Lake

Apache Airflow

Cloud platforms

Job description

We are working with a highly successful, technology-driven organisation that is looking to hire an experienced Software Engineer (open to python or C#) to join its Data Engineering team in Dublin.

This is an opportunity to work in a technically challenging, engineering-led environment where software quality, performance, scalability and reliability are central to everything the team builds.

The successful candidate will design, develop and maintain robust data processing systems, working with large datasets and modern distributed data technologies.

This is a hands-on software engineering position rather than a traditional analytics or reporting-focused data engineering role. We are particularly interested in engineers who enjoy writing high-quality production code, solving complex technical problems and building reliable data infrastructure.

What You'll Be Doing

• Design, develop and maintain scalable data pipelines and data processing applications.
• Build robust, production-quality software primarily using Python.
• Work with large datasets, relational databases and distributed processing systems.
• Develop and optimise batch processing workflows, ensuring reliability, performance and data integrity.
• Improve the efficiency, scalability and maintainability of existing data infrastructure.
• Work with modern cloud-based data technologies and distributed computing frameworks.
• Collaborate with experienced software engineers and other technical stakeholders to deliver high-quality solutions.
• Take ownership of your code throughout the software development lifecycle, including testing, deployment, monitoring and ongoing improvements.
• Approximately 3-5+ years of professional software engineering experience building and maintaining production systems.
• Strong hands-on Python development experience within substantial, production-level codebases.
• Professional experience with at least one additional programming language, such as Java, C# or Scala.
• Strong SQL skills and a solid understanding of relational database systems.
• Experience designing, building or maintaining batch data processing pipelines.
• Good understanding of software engineering fundamentals, including data structures, algorithms, testing and system design.
• Experience troubleshooting, optimising and supporting production applications.
• A strong problem-solving mindset and an interest in building efficient, reliable software.

Highly Desirable

Experience with one or more of the following technologies or concepts:

• Databricks
• Delta Lake or Apache Hudi
• Apache Airflow or similar workflow orchestration tools
• Distributed data processing and large-scale ETL/ELT systems
• Cloud-based data platforms and infrastructure

Why Consider This Opportunity?

• Interesting engineering challenges: Work on complex data processing problems involving performance, scalability and reliability.
• Modern technologies: Gain exposure to distributed computing, modern data platforms and large-scale data infrastructure.
• Strong engineering culture: Collaborate with technically accomplished engineers who value clean code, sound architecture and practical problem-solving.
• Real ownership: Contribute directly to the design, implementation and ongoing development of production systems.
• Long-term technical growth: Develop deeper expertise in software engineering, distributed systems and modern data technologies.