Senior Data Engineer: Scalable Pipelines & Real-Time Data

Prudential Annuities Distributors (PAD); Ireland; Remote; EUR 90,000 - 130,000; Full time

Job summary

Prudential Ireland, part of PGIM, is seeking a Senior Data Engineer to partner with architects, data scientists and analysts to design, develop and maintain high-quality data pipelines. You will work on data ingestion, transformation and analytics, utilizing Spark, Kinesis, Kafka and a broad set of AWS services, with a strong emphasis on security, scalability and collaboration.

You will contribute to developing data models, ensure data quality, and drive platform delivery through an agile,

Qualifications

• Bachelor of Computer Science or Engineering or related field.
• Experience in DevOps automation tools & practices.
• Knowledge of full software development life cycle (SDLC).
• Ability to learn new skills and knowledge on an ongoing basis.
• Excellent problem solving, communication and collaboration skills.
• Applied experience with Python, R, SQL, Java, Scala, Pyspark/Apache Spark, Shell scripting.
• Data Ingestion, Integration & Transformation across multiple sources.
• Database management across relational, NoSQL, graph DBs and data warehouses including Snowflake/Redshift.
• Real-Time Analytics with Spark and streaming technologies.
• Data buffering with Kinesis and Kafka.
• Workflow orchestration with Airflow and related tools.
• Data visualization with Tableau and Power BI.
• Data Lakes and warehousing design and governance.
• Data protection, security, and access controls.

Responsibilities

• Build and optimize data pipelines, logic and storage systems; code against the roadmap.
• Develop high quality, well documented code; adhere to Prudential standards.
• Conduct complex data analysis and prepare data for modeling.
• Collaborate with data analysts, scientists, and architects on data projects.
• Write unit, integration and functional tests; address issues raised by QA.
• Embed learning and innovation; identify opportunities to improve processes.
• Tackle complex problems requiring in-depth analysis of multiple factors.
• Use languages like Python, R, SQL, Java, Scala, Pyspark/Apache Spark, Shell scripting.

Skills

Python

R

SQL

Java

Scala

Pyspark

Shell scripting

DevOps

Data pipelines

Spark

Kinesis

Kafka

Airflow

CloudFormation

Education

Bachelor of Computer Science or Engineering

Tools

AWS

Snowflake

Tableau

Power BI

MS Excel

Job description

Prudential Ireland, part of PGIM, is seeking a Senior Data Engineer to partner with architects, data scientists and analysts to design, develop and maintain high-quality data pipelines. You will work on data ingestion, transformation and analytics, utilizing Spark, Kinesis, Kafka and a broad set of AWS services, with a strong emphasis on security, scalability and collaboration.

You will contribute to developing data models, ensure data quality, and drive platform delivery through an agile,