Senior Data Engineer

Stefanini Group; Contagem; Presencial; BRL 80.000 - 120.000; Tempo integral

Resumo da oferta

A prominent technology firm based in Brazil seeks a skilled professional to manage and optimize data pipelines in AWS environments. This role entails creating and maintaining data structures, ensuring compliance with data governance, and effectively collaborating with stakeholders. Ideal candidates will possess strong experience in data management and modeling, excellent communication ability, and technical knowledge in tools like Databricks, Spark, and Python. Remote work options are available.

Qualificações

• Strong experience with various Data Management architectures.
• Ability to design and optimize data pipelines.
• Experience with commercial and open-source message queuing technologies.

Responsabilidades

• Create and optimize data pipelines.
• Drive Automation through metadata management.
• Collaborate with data owners and stakeholders.

Conhecimentos

Data Modeling

AWS environment

Collaboration

Communication skills

Creativity

Ferramentas

Databricks

Spark

Python

Terraform

SQL

Scala

AWS Glue

AWS Athena

Descrição da oferta de emprego

Mission

• Responsible for building, managing, and optimizing data pipelines;
• Support for defining data flows and access to data sources;

Ensure compliance with data governance, data architecture, and data security requirements.

Responsibilities

• Create, maintain, and optimize data pipelines for data structures that encompass data modeling, data transformation, schemas, metadata, and workload management;
• Drive Automation through effective metadata management;
• Participate in ensuring compliance and governance during data use, including the maintenance of the data catalog;
• Collaborate and support data owners and other stakeholders around data topics;
• Ensure the proper use of the data architecture defined by Data Factory team;
• Respect and follow security principles ensuring compliance with data delivery.

Competencies & Skills

• Strong experience with various Data Management architectures, mainly AWS environment;
• Strong experience with Data Modeling;
• Ability to design, build and manage data pipelines;
• Experience working with commercial and open source message queuing technologies;
• Highly creative and collaborative;
• Excellent communication skills and the ability to collaborate effectively with cross‑functional teams.

Specific Technical Knowledge

• Solid knowledge in: Databricks, Spark, python, terraform, SQL, Scala, S3, AWS Glue, AWS Athena, AWS Lambda, AWS Step Functions; Big data, Relational Databases;
• Experience in: EMR, Gitflow, Agile Metodology; DevOps.

Languages

English: Advanced or Fluent

Additional information

Remote work