Azure Data Engineer: Build Scalable Data Platforms

Affinity; Setúbal; Híbrido; EUR 50 000 - 75 000; Tempo integral

Resumo da oferta

Affinity, a Portuguese technology consulting company, is seeking a Data Engineer to design, build, and maintain scalable data platforms. You will collect and process large datasets, implement data workflows with Spark and Apache Beam, and collaborate across teams to ensure data integrity and security.

Hybrid work setup in Lisbon or Porto, with exposure to leading Big Data technologies, cloud platforms, and modern ETL practices.

Qualificações

• Experience with cloud platforms and services for data engineering (Azure)
• Proficiency in programming languages like Python, Java, or Scala
• Use of Big Data tools such as Spark, Flink, Kafka, ElasticSearch, Hadoop, Hive, Sqoop, Flume, Impala, Kafka Streams and Connect, Druid
• Knowledge of data modeling and database design principles
• Familiarity with data integration and ETL tools (Apache Kafka, Talend)
• Understanding of distributed systems and data processing architectures
• Strong SQL skills and experience with relational and NoSQL databases
• Familiarity with cloud platforms for data engineering (AWS S3, Azure Data Factory)
• Experience with version control tools such as Git

Responsabilidades

• Design, build, and maintain scalable data platforms
• Collect, process, and analyze large and complex data sets from various sources
• Develop and implement data processing workflows using Spark and Apache Beam
• Collaborate with cross-functional teams to ensure data accuracy and integrity
• Ensure data security and privacy through proper access controls and data encryption
• Extraction of data from various sources, including databases, file systems, and APIs
• Monitor system performance and optimize for high availability and scalability

Conhecimentos

Python

Java

Scala

SQL

Ferramentas

Spark

Kafka

Hadoop

Hive

Talend

Git

Flink

Beam

Druid

Descrição da oferta de emprego

Affinity, a Portuguese technology consulting company, is seeking a Data Engineer to design, build, and maintain scalable data platforms. You will collect and process large datasets, implement data workflows with Spark and Apache Beam, and collaborate across teams to ensure data integrity and security.

Hybrid work setup in Lisbon or Porto, with exposure to leading Big Data technologies, cloud platforms, and modern ETL practices.