Data Engineer

Pyramid Consulting, Inc; Carapicuíba; Presencial; BRL 80.000 - 120.000; Tempo integral

Resumo da oferta

A consulting firm is seeking a skilled data engineer with expertise in cloud platforms such as AWS, modern data architectures, and proficiency in tools like Kafka and AWS Glue. Candidates should have a strong SQL background and experience with DevOps practices. This role is crucial for the development of robust data pipelines and orchestration using various frameworks. Proficient knowledge in Python and streaming architectures is essential.

Conhecimentos

Familiarity with major cloud platforms (AWS)

Experience with modern data architectures

Proficiency in ingestion tools (Kafka, AWS Glue)

Knowledge of streaming architectures and tools (Kafka, Flink)

Strong background in SQL-based frameworks

Experience with SCD and schema evolution

Familiarity with Spark for transformation

Proficient in Python

Demonstrated experience in DevOps practices

Understanding of governance principles

Descrição da oferta de emprego

Familiarity with major cloud platforms (AWS)Experience with modern data architectures (data lakes, lakehouses, hubs)Proficiency in ingestion tools (Kafka, AWS Glue) and formats (Iceberg, Parquet)Knowledge of streaming architectures and tools (Kafka, Flink)Strong background in SQL-based frameworks and orchestration tools (dbt, Glue, Airflow).

Experience with SCD and schema evolutionFamiliarity with Spark for transformation, streaming, tuning, debugging with Spark UIProficient in Python.

Expert-level SQLDemonstrated experience in DevOps practices (code management, CI / CD, deployment)Understanding of governance principles (quality, privacy, security)