Senior Data Engineer

Stefanini Group; Teletrabalho; BRL 60.000 - 100.000; Tempo integral

Resumo da oferta

A technology consulting firm is seeking a professional to manage and optimize data pipelines within an AWS environment. The role necessitates strong experience in data management architectures and data modeling, and excellent communication skills to collaborate with cross-functional teams. Candidates should be proficient in tools like Databrics, Spark, and Terraform, and must possess advanced or fluent English. The position supports remote work, providing flexibility to the employee.

Qualificações

• Strong experience with various Data Management architectures, mainly in AWS.
• Ability to design, build, and manage data pipelines.
• Advanced or fluent English communication skills.

Responsabilidades

• Build, manage, and optimize data pipelines.
• Ensure compliance with data governance and security requirements.
• Collaborate with stakeholders regarding data topics.

Conhecimentos

Data Management architectures

Data Modeling

Designing data pipelines

Message queuing technologies

Creative collaboration

Effective communication

Ferramentas

Databrics

Spark

Python

Terraform

SQL

Scala

S3

AWS Glue

AWS Athena

AWS Lambda

AWS Step Functions

Big Data

Relational Databases

EMR

Gitflow

Agile Methodology

DevOps

Descrição da oferta de emprego

Mission

• Responsible for building, managing, and optimizing data pipelines;
• Support for defining data flows and access to data sources;
• Ensure compliance with data governance, data architecture, and data security requirements.

Responsibilities

• Create, maintain, and optimize data pipelines for data structures that encompass data modeling, data transformation, schemas, metadata, and workload management;
• Drive Automation through effective metadata management;
• Participate in ensuring compliance and governance during data use, including the maintenance of the data catalog;
• Collaborate and support data owners and other stakeholders around data topics;
• Ensure the proper use of the data architecture defined by Data Factory team;
• Respect and follow security principles ensuring compliance with data delivery.

Competencies & Skills

• Strong experience with various Data Management architectures, mainly AWS environment;
• Strong experience with Data Modeling;
• Ability to design, build and manage data pipelines;
• Experience working with commercial and open source message queuing technologies;
• Highly creative and collaborative;
• Excellent communication skills and the ability to collaborate effectively with cross-functional teams.

Specific Technical Knowledge

• Solid knowledge in : Databrics, Spark, python, terraform, SQL, Scala, S3, AWS Glue, AWS Athena, AWS Lambda, AWS Step Functions; Big data, Relational Databases;
• Experience in : EMR, Gitflow, Agile Metodology; DevOps.

Languages

• English : Advanced or Fluent

Additional information

• Remote work