Senior Data Engineer | PySpark, Databricks & Azure

United States Digital Space LLC; Dublin; Hybrid; EUR 85,000 - 140,000; Full time

Benefits offered by this job

International projects

Hybrid or remote flexibility

Medical healthcare

Recognition program

Ongoing learning & reimbursement

Well-being program

Team events & local benefits

Sports compensation

Referral bonuses

Top-tier equipment provision

Job summary

United States Digital Space LLC is seeking a Senior Data Engineer to turn data into actionable insights. You will design, build, and optimize Spark-based data pipelines using PySpark on Databricks, write high-quality Python code, and collaborate with cross-functional teams to deliver scalable data solutions.

The role requires strong SQL, data modeling, and experience with Delta Lake and Parquet formats. You will work with a global enterprise client and contribute to cloud-native data platforms.

Qualifications

• Solid commercial experience in data engineering.
• Practical experience developing SQL stored procedures and migrating them into Spark SQL or PySpark environments.
• Extensive expertise using PySpark on Databricks, encompassing Delta tables, cluster management, and workflow automation.
• Proven track record of optimizing Spark performance within production-level projects.
• Strong proficiency in Python, specifically for complex data manipulation utilizing libraries like Polars or Pandas.
• Solid understanding of columnar data storage formats, particularly Parquet, with practical experience using Delta Tables.
• Proven expertise in data processing, analysis, and transformation workflows.
• Strong analytical and problem-solving abilities with a detail-oriented mindset.

Responsibilities

• Define and enforce best practices and coding standards across the project.
• Conduct thorough code reviews to ensure adherence to established guidelines and maintain high code quality.
• Work independently and with the team to deliver robust Spark-based solutions.
• Communicate instructions to team members and help manage day-to-day operations.
• Regularly communicate with the client and stakeholders.
• Design, develop, and maintain scalable Spark applications.
• Write clean, maintainable code following best practices.
• Optimize code for performance and scalability with efficient data handling.
• Collaborate with cross-functional teams to deliver high-quality software solutions.
• Identify and resolve technical issues ensuring reliability and performance.
• Create and maintain comprehensive documentation for code, processes, and workflows.

Skills

Python

PySpark

Spark SQL

SQL

Data modeling

Tools

Databricks

Delta Lake

Parquet

Azure Cloud

Job description

United States Digital Space LLC is seeking a Senior Data Engineer to turn data into actionable insights. You will design, build, and optimize Spark-based data pipelines using PySpark on Databricks, write high-quality Python code, and collaborate with cross-functional teams to deliver scalable data solutions.

The role requires strong SQL, data modeling, and experience with Delta Lake and Parquet formats. You will work with a global enterprise client and contribute to cloud-native data platforms.