Automotive Data Engineer — Scalable Real-Time Pipelines

IRIUM Portugal; Lisboa; On-site; EUR 48,000 - 75,000; Full time

Benefits offered by this job

Medical insurance

Flexible pay

Technical training access

Job summary

IRIUM Portugal is seeking a Data Engineer to join a vehicle manufacturers project. You will upload data to the big data platform, prepare data profiles, and develop ready-to-use data pipelines to support multiple use cases.

You will collaborate with teams and, for complex tasks, coordinate service providers. A strong background in distributed systems, data processing, and automation is essential for success.

Qualifications

• Proficient in programming languages Apache Spark, Python, R, Scala, Java and Kotlin.
• Understanding of AWS Cloud Ingestion solutions.
• Experience with distributed data processing and S3/databases (on-prem and cloud).
• Knowledge of source systems and their data types and structures.
• Experience with batch and real-time interfaces implementation.
• Expertise in building data products with testing, deployment, logging, monitoring, and operations.
• Experience with Docker-based deployment and Resource Manager.
• Experience with CI/CD pipelines and tools.
• Knowledge of Workflow Managers and Scrum-based Agile development.
• Technical knowledge of the automotive industry.
• 3 years of experience.

Responsibilities

• Uploading data to the Big Data Platform
• Targeted data preparation for specific use cases and creation of data profiles
• Development of ready-to-use data pipelines
• Coordination of the service provider's activities (only complex)

Skills

Apache Spark

Python

R

Scala

Java

Kotlin

Tools

Docker deployment

Resource Manager

Workflow Managers

CI/CD tools

AWS Cloud Ingestion

S3

Databases

Job description

IRIUM Portugal is seeking a Data Engineer to join a vehicle manufacturers project. You will upload data to the big data platform, prepare data profiles, and develop ready-to-use data pipelines to support multiple use cases.

You will collaborate with teams and, for complex tasks, coordinate service providers. A strong background in distributed systems, data processing, and automation is essential for success.