Senior Data Engineer, Data Lake

Pismo Soluções Tecnologicas Ltda.; São Paulo; Remoto; BRL 120.000 - 180.000; Tempo pieno

Descrizione del lavoro

Visa is seeking a Data Engineer Analyst to join the Data Lake team. You will build and maintain ingestion and transformation pipelines using Spark, Databricks, and Airflow to ensure a reliable, high‑quality corporate data lake for decision‑making both internally and for clients.

In this intermediate role, you’ll work under moderate supervision, expand your expertise in lakehouse architecture, and contribute to scalable data infrastructure while aligning with governance standards and best

Competenze

• 3+ years of relevant work experience with a Bachelor's or Associate’s Degree OR 5+ years of relevant work experience.
• Python for automation and data processing
• Data pipeline concepts (batch, ETL/ELT patterns)

Mansioni

• Develop, test, and maintain data pipelines (ingestion, transformation, quality checks) using PySpark/SparkSQL on Databricks.
• Build and modify Airflow DAGs (MWAA) for pipeline orchestration.
• Write and optimize SQL queries for data transformation and validation.
• Support data quality by implementing and monitoring quality checks (Great Expectations or equivalent).
• Participate in code reviews — both giving and receiving feedback.
• Investigate pipeline failures and data quality issues with guidance from senior engineers.
• Write and maintain documentation for datasets and pipelines you build.
• Participate in sprint planning, estimation, and retrospectives.
• Follow CI/CD, testing, and governance standards defined by the team.

Conoscenze

Python

Data pipelines

ETL/ELT concepts

Spark

Databricks

Airflow

SQL

Git/GitHub

Formazione

Bachelor's degree or Associate’s degree

Strumenti

Databricks

Airflow

Spark

SQL

AWS S3

Git/GitHub

Descrizione del lavoro

About Us

Visa is a world leader in payments technology, facilitating transactions between consumers, merchants, financial institutions and government entities across more than 200 countries and territories, dedicated to uplifting everyone, everywhere by being the best way to pay and be paid.

About Us

Visa is a world leader in payments technology, facilitating transactions between consumers, merchants, financial institutions and government entities across more than 200 countries and territories, dedicated to uplifting everyone, everywhere by being the best way to pay and be paid.

At Visa, you'll have the opportunity to create impact at scale — tackling meaningful challenges, growing your skills and seeing your contributions impact lives around the world.

Join Visa and do work that matters – to you, to your community, and to the world. Progress starts with you.

Job Description

We're looking for a Data Engineer at the Analyst level to join our Data Lake team. You'll build and maintain data ingestion and transformation pipelines using Spark, Databricks, and Airflow, contributing to the reliability and quality of the corporate data lake. This is an intermediate role where you'll work under moderate supervision on defined tasks while building deeper expertise in lakehouse architecture.

At Pismo, the Data Lake team is responsible for centralizing and organizing data into a single, trusted platform that supports decision-making across the company and for external clients. We work on challenges such as scaling global data infrastructure, delivering high-quality reporting, and enabling secure, self-service access to data—helping teams move faster while avoiding information silos.

What You'll Do

• Develop, test, and maintain data pipelines (ingestion, transformation, quality checks) using PySpark/SparkSQL on Databricks.
• Build and modify Airflow DAGs (MWAA) for pipeline orchestration.
• Write and optimize SQL queries for data transformation and validation.
• Support data quality by implementing and monitoring quality checks (Great Expectations or equivalent).
• Participate in code reviews — both giving and receiving feedback.
• Investigate pipeline failures and data quality issues with guidance from senior engineers.
• Write and maintain documentation for datasets and pipelines you build.
• Participate in sprint planning, estimation, and retrospectives.
• Follow CI/CD, testing, and governance standards defined by the team.

This is a remote position. A remote position does not require job duties be performed within proximity of a Visa office location. Remote positions may be required to be present at a Visa office with scheduled notice. Remote

Qualifications

For this role, you must be based in Brazil

Language Skills

• Proficiency in English at B2 level or above (Upper-Intermediate)

Basic Qualifications

• 3+ years of relevant work experience with a Bachelor's or Associate’s Degree OR 5+ years of relevant work experience.
• Python for automation and data processing
• Data pipeline concepts (batch, ETL/ELT patterns)

Technical Skills

• Apache Spark basics (PySpark or SparkSQL)
• Databricks (jobs, workflows, cluster management, tuning)
• SQL (intermediate–advanced: joins, window functions, CTEs)
• Amazon S3 data lake design (partitioning, layout, lifecycle)
• Basic cloud concepts (AWS: S3, IAM, CloudWatch)
• Data modeling (dimensional / Kimball, medallion layers)
• Git/GitHub workflow and code review

Preferred Qualifications

• Delta Lake (ACID tables, OPTIMIZE, VACUUM, schema evolution, MERGE)
• Apache Airflow / MWAA (DAG design, retries, backfills, SLAs)
• CDC patterns (DMS, incremental processing, MERGE upserts)
• Data quality concepts (validation, testing)
• Terraform basics
• BI integration (Superset, dashboarding)

Visa is an EEO Employer

Qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, sexual orientation, gender identity, disability or protected veteran status. Visa will also consider for employment qualified applicants with criminal histories in a manner consistent with EEOC guidelines and applicable local law.