Data Engineer – Specialist

Jobtailor; São Paulo; Presencial; BRL 120 000 - 210 000; Tempo integral

Resumo da oferta

Jobtailor in São Paulo seeks a Data Engineer to build and maintain data pipelines on Google Cloud Platform, using BigQuery, Dataflow, Pub/Sub, and Vertex AI. The role focuses on Python-based ETL development, API integrations, and preprocessing data for ML/AI training and inference.

The ideal candidate will have experience with Airflow/Cloud Composer, strong SQL and Python skills, and a solid background in data analysis and pipeline orchestration. Excellent communication is welcome.

Qualificações

• Education: Bachelor's degree in Computer Science, Engineering or related fields.
• Experience developing and using Airflow (preferably Cloud Composer on GCP).
• Proven software development experience, preferably with public projects on GitHub or similar.
• Strong knowledge of SQL and Python for ETL development.
• Experience with GCP services: BigQuery, Dataflow, Dataproc, Cloud Storage.

Responsabilidades

• Develop Airflow DAGs (Cloud Composer) for data ETL pipelines.
• Develop scripts for data ETL in Dataflow (Java), Dataproc (Python) and BigQuery (SQL).
• Software development to support ETL processes, primarily using Python.
• Develop and maintain data pipelines for training and inference of ML/AI models on Vertex AI.
• Integrate APIs and process files in multiple formats (JSON, CSV, Parquet, Avro, etc.).
• Evaluate and adopt AI tools to increase data engineering productivity.

Conhecimentos

Airflow Development

Python for ETL

SQL

Data Analysis

Cloud Knowledge

Formação académica

Bachelor's degree in Computer Science, Engineering or related fields

Ferramentas

Cloud Composer

BigQuery

Dataflow

Dataproc

Pub/Sub

Vertex AI

Descrição da oferta de emprego

• 100% environment on Google Cloud Platform (GCP): BigQuery, Dataflow, Dataproc, Cloud Storage, Pub/Sub, Vertex AI;
• Orchestration with Cloud Composer (managed Airflow on GCP);
• Development of Airflow DAGs (Cloud Composer) for data ETL using Python;
• Development of scripts for data ETL in Dataflow (Java), Dataproc (Python) and BigQuery (SQL);
• Software development to support ETL processes, primarily using Python;
• Development and integration of data pipelines with external APIs and processing of files in multiple formats (JSON, CSV, Parquet, Avro, etc.);
• Development and maintenance of data pipelines for training and inference of Machine Learning and AI models on Vertex AI;
• Ability to perform data analysis to identify and resolve bugs in ETL processes;
• Experience with feature engineering and preparing datasets for ML/AI models;
• Critical evaluation of outputs from generative AI tools applied to data (LLMs, data agents, etc.);
• Knowledge of workflow orchestration with AI agents for automating data processes;
• Experience or interest in integrating LLMs into data pipelines (e.g., enrichment, classification, entity extraction);
• Ability to evaluate, adopt and adapt AI tools to increase data engineering productivity;
• Develop and maintain effective communication, facilitating interaction between Dotz’s Engineering, Data and Business teams.

Requirements

• Education: Bachelor's degree in Computer Science, Engineering or related fields;
• Experience developing and using Airflow (preferably Cloud Composer on GCP);
• Proven software development experience, preferably with public projects on GitHub or similar;
• Strong knowledge of SQL and Python for ETL development;
• Experience with GCP services: BigQuery, Dataflow, Dataproc, Cloud Storage;
• Proven experience in data analysis, preferably in a corporate environment;
• Experience integrating APIs and processing files in multiple formats;
• Familiarity with ML/AI concepts applied to data (feature engineering, training pipelines, inference);
• Experience or knowledge of Vertex AI or equivalent cloud ML platforms;
• Ability to use generative AI tools (LLMs, agents, copilots) to boost day-to-day productivity;
• Excellent verbal and written communication skills;
• Ability to facilitate discussions between different areas and unblock critical issues;

Core Competencies

Demonstrates expertise in developing and maintaining data pipelines on Google Cloud Platform, utilizing tools such as BigQuery, Dataflow, and Cloud Composer. Proficient in Python and SQL for ETL processes, with a strong understanding of machine learning concepts and the integration of AI tools to enhance data engineering productivity.

Highest-signal resume keywords

• Google Cloud Platform (GCP)
• Airflow Development
• Python for ETL
• Data Analysis
• Machine Learning Integration

ATS Optimization Keywords

Hard Skills

• BigQuery
• Dataflow
• Dataproc
• Cloud Storage
• SQL
• Python
• ETL Development
• API Integration
• Feature Engineering
• Vertex AI

Soft Skills

• Excellent Communication Skills
• Facilitation Skills

Industry Keywords

• Data Engineering
• Machine Learning
• AI Tools
• Data Analysis
• Workflow Orchestration

Tools & Technologies

• Cloud Composer
• Pub/Sub
• Data Pipelines
• Generative AI Tools