Data Engineer

Axiom Software Solutions Limited; Barcelona; Presencial; EUR 30.000 - 50.000; Jornada completa

Descripción de la vacante

An innovative firm is on the lookout for a talented Data Engineer with a strong foundation in PySpark and AWS services. This exciting role involves designing and optimizing large-scale data processing systems, ensuring efficient data flow and transformation. You'll collaborate with data scientists and analysts to deliver tailored solutions, while also focusing on data quality and security. Join a dynamic environment where your analytical skills and problem-solving abilities can shine, making a real impact on data-driven decision-making. If you're ready to take on new challenges and grow your career in data engineering, this opportunity is perfect for you.

Formación

• 5+ years of experience as a Data Engineer with a focus on PySpark.
• Proficient in AWS services including Athena and EMR.

Responsabilidades

• Design and maintain scalable data pipelines using PySpark.
• Implement data quality and governance best practices.

Conocimientos

PySpark

AWS (Athena, EMR)

SQL

Python

Analytical Skills

Communication Skills

Problem-Solving

Agility

Educación

Bachelor's degree in Computer Science

Master’s degree in Engineering

Herramientas

Apache Airflow

Hadoop

Hive

Kafka

AWS (S3, Lambda, Redshift)

Descripción del empleo

Job Title: Data Engineer

Job Description:

Seeking a skilled Data Engineer with a robust background in PySpark and extensive experience with AWS services, including Athena and EMR. The ideal candidate will be responsible for designing, developing, and optimizing large-scale data processing systems, ensuring efficient and reliable data flow and transformation.

Key Responsibilities:

1. Data Pipeline Development: Design, develop, and maintain scalable data pipelines using PySpark to process and transform large datasets.
2. AWS Integration: Utilize AWS services, including Athena and EMR, to manage and optimize data workflows and storage solutions.
3. Data Management: Implement data quality, data governance, and data security best practices to ensure the integrity and confidentiality of data.
4. Performance Optimization: Optimize and troubleshoot data processing workflows for performance, reliability, and scalability.
5. Collaboration: Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions that meet business needs.
6. Documentation: Create and maintain comprehensive documentation of data pipelines, ETL processes, and data architecture.

Required Skills and Qualifications:

1. Education: Bachelor's or Master’s degree in Computer Science, Engineering, or a related field.
2. Experience: 5+ years of experience as a Data Engineer or in a similar role, with a strong emphasis on PySpark.
3. Technical Expertise:
1. Proficient in PySpark for data processing and transformation.
2. Extensive experience with AWS services, specifically Athena and EMR.
3. Strong knowledge of SQL and database technologies.
4. Experience with Apache Airflow is a plus.
5. Familiarity with other AWS services such as S3, Lambda, and Redshift.
4. Programming: Proficiency in Python; experience with other programming languages is a plus.
5. Problem-Solving: Excellent analytical and problem-solving skills with attention to detail.
6. Communication: Strong verbal and written communication skills to effectively collaborate with team members and stakeholders.
7. Agility: Ability to work in a fast-paced, dynamic environment and adapt to changing priorities.

Preferred Qualifications:

1. Experience with data warehousing solutions and BI tools.
2. Knowledge of other big data technologies such as Hadoop, Hive, and Kafka.
3. Understanding of data modeling, ETL processes, and data warehousing concepts.
4. Experience with DevOps practices and tools for CI/CD.