Senior Data Engineer - Spark & Databricks (Remote)

United States Digital Space LLC; Dublin; On-site; EUR 90,000 - 120,000; Full time

Benefits offered by this job

International projects

In-office, hybrid, or remote flexibill

Medical healthcare

Recognition program

Learning reimbursement

Well-being program

Team events & local benefits

Sports compensation

Referral bonuses

Top-tier equipment

Job summary

the company in Ireland seeks a Senior Data Engineer to design and implement data solutions using PySpark, Databricks, and Spark SQL. You will build robust data pipelines, optimize performance, and collaborate across teams to deliver scalable software.

The role requires hands-on experience with Python data processing, Delta Tables, and cloud-based Databricks environments. You will interface with clients and contribute to architectural decisions in a fast-paced enterprise setting.

Qualifications

• Experience developing Spark/Databricks data pipelines.
• Expert in Python for data manipulation with Pandas/Polars.
• Ability to migrate SQL to Spark SQL or PySpark.
• Proven track record optimizing Spark performance in production.
• Strong collaboration and communication with clients.

Responsibilities

• Define and enforce best practices and coding standards.
• Conduct thorough code reviews to ensure high quality.
• Work independently and in collaboration with the team.
• Communicate instructions to team members and manage day-to-day operations.
• Maintain regular communication with the client.
• Design, develop, and maintain Spark applications.
• Write clean, efficient code following best practices.
• Optimize code for performance and scalability.
• Work with cross-functional teams to deliver software solutions.
• Identify and resolve issues to ensure reliability and performance.
• Create and maintain documentation for code and workflows.

Skills

Python

PySpark

Databricks

Spark SQL

SQL Stored Procedures

Delta Tables

Polars

Azure Cloud

Data pipelines

Microservices

Tools

Docker

Kubernetes

GitHub Enterprise

LangGraph

Job description

the company in Ireland seeks a Senior Data Engineer to design and implement data solutions using PySpark, Databricks, and Spark SQL. You will build robust data pipelines, optimize performance, and collaborate across teams to deliver scalable software.

The role requires hands-on experience with Python data processing, Delta Tables, and cloud-based Databricks environments. You will interface with clients and contribute to architectural decisions in a fast-paced enterprise setting.