Data Engineer – Python, PySpark & AWS

Avenue Code

Brasil

On-site

BRL 180,000 - 300,000

Full time

11 days ago

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Avenue Code is seeking a skilled Data Engineer to design, develop, and optimize scalable data pipelines in an AWS cloud environment. You will build ETL/ELT processes, integrate multiple data sources, and develop data lakes and warehouses using Python, SQL, PySpark, and AWS services.

The role spans the full data engineering lifecycle, from requirements gathering to implementation, testing, and handoff, with strong emphasis on data quality and reliability in a collaborative team setting.

Qualifications

  • Solid professional experience in Data Engineering with production data pipelines.
  • Strong proficiency in Python for data processing, transformation, and automation.
  • Advanced SQL skills with complex queries and performance optimization.
  • Hands-on experience with PySpark and distributed data processing.
  • Experience designing, developing, and maintaining ETL/ELT pipelines.
  • Practical experience with AWS services: S3, Glue, Athena, Lambda, Redshift.
  • Understanding of Data Lake and Data Warehouse architectures.
  • Experience with data modeling and data integration across sources.
  • Familiarity with data-quality validation, monitoring, and troubleshooting.
  • Experience with Git and CI/CD; Terraform familiarity.

Responsibilities

  • Design, develop, test, and maintain scalable ETL/ELT pipelines across multiple data sources.
  • Build and optimize data-processing solutions using Python, SQL, and PySpark.
  • Develop cloud-native data solutions with AWS services (S3, Glue, Athena, Lambda, Redshift).
  • Design and maintain Data Lakes and Data Warehouses for efficient storage and access.
  • Transform data models and workflows to support analytics and BI requirements.
  • Integrate structured and semi-structured data ensuring quality and reliability.
  • Implement data-quality checks and monitoring across pipelines.
  • Collaborate with stakeholders to translate data needs into technical solutions.
  • Participate in design discussions on data architecture and patterns.
  • Maintain reusable, well-structured code following best practices.
  • Work with Git, CI/CD, and Terraform for deployments and infra management.
  • Troubleshoot pipeline failures and deliver long-term fixes.
  • Produce documentation covering architecture and procedures.
  • Take end-to-end ownership of data engineering deliverables.

Skills

Python
SQL
PySpark
Data pipelines
ETL/ELT
Data modeling

Education

Degree in Computer Science / Data Engineering

Tools

AWS
S3
Glue
Athena
Lambda
Redshift

Job description

About The Opportunity

We are looking for a skilled and experienced

About The Opportunity

We are looking for a skilled and experienced Data Engineer to join our team and contribute to the design, development, and optimization of scalable data solutions in an AWS cloud environment. In this role, you will be responsible for building and maintaining robust ETL/ELT pipelines, integrating data from multiple sources, and developing efficient data-processing solutions that support business operations and data-driven decision-making. You will work with modern cloud-based data technologies, leveraging Python, SQL, PySpark, and AWS services to build reliable data pipelines, Data Lakes, and Data Warehouses. You will collaborate with cross-functional teams to understand business requirements, implement technical solutions, and ensure data quality, performance, and reliability. The ideal candidate is a hands‑on data engineer with strong expertise in Python, SQL, PySpark, AWS Glue, S3, Athena, Lambda, and Redshift, who is comfortable working across the entire data engineering lifecycle, from requirements gathering and solution design to implementation, testing, troubleshooting, documentation, and technical handoff.

Responsibilities
  • Design, develop, test, and maintain scalable ETL/ELT pipelines to support data integration, transformation, and processing across multiple data sources.
  • Build and optimize data-processing solutions using Python, SQL, and PySpark.
  • Develop and maintain cloud-native data solutions using AWS services, including S3, Glue, Athena, Lambda, and Redshift.
  • Design, implement, and support Data Lakes and Data Warehouses, ensuring efficient data storage, organization, accessibility, and processing.
  • Develop data models and transformation workflows to support analytics, reporting, and business intelligence requirements.
  • Integrate structured and semi-structured data from multiple sources, ensuring consistency, accuracy, and reliability.
  • Implement data-quality checks, validation processes, and monitoring mechanisms to maintain data integrity across pipelines.
  • Optimize data pipelines, SQL queries, and distributed processing workloads to improve performance, scalability, and cost efficiency.
  • Collaborate with business stakeholders, analysts, and engineering teams to understand data requirements and translate them into effective technical solutions.
  • Participate in technical design discussions, contributing to data architecture, integration patterns, and engineering best practices.
  • Develop and maintain reusable, well-structured, and maintainable code following data engineering best practices.
  • Work with Git, CI/CD pipelines, and Terraform to support version control, automated deployments, and infrastructure management.
  • Troubleshoot complex data-processing issues, investigate pipeline failures, and implement reliable, long-term solutions.
  • Perform testing, validation, and debugging to ensure the quality and reliability of data solutions.
  • Produce technical documentation covering pipeline architecture, data transformations, implementation decisions, and operational procedures.
  • Take end-to-end ownership of data engineering deliverables, from requirements analysis and development through testing, implementation, and technical handoff.
Required Qualifications
  • Solid professional experience in Data Engineering, with a strong background in developing and maintaining production data pipelines.
  • Strong proficiency in Python for data processing, transformation, and automation.
  • Advanced SQL skills, including complex queries, data transformations, performance optimization, and troubleshooting.
  • Hands‑on experience with PySpark and distributed data processing.
  • Strong experience designing, developing, and maintaining ETL/ELT pipelines.
  • Practical experience with AWS cloud services, particularly Amazon S3, AWS Glue, Amazon Athena, AWS Lambda, and Amazon Redshift.
  • Solid understanding of Data Lake and Data Warehouse architectures, including data storage, processing, and integration patterns.
  • Experience with data modeling, data transformation, and data integration across multiple sources.
  • Familiarity with data-quality validation, monitoring, and pipeline troubleshooting.
  • Experience with Git and collaborative development workflows.
  • Familiarity with CI/CD practices and Terraform for automated deployments and infrastructure management.
  • Strong understanding of data engineering principles, code quality, performance optimization, and maintainability.
  • Experience working across the full data engineering lifecycle, including requirements gathering, implementation, testing, troubleshooting, documentation, and technical handoff.
  • Strong analytical and problem‑solving skills, with the ability to independently investigate and resolve complex technical issues.
  • Excellent communication and collaboration skills.
  • Ability to work independently, manage priorities, and deliver high‑quality solutions in a collaborative environment.
Nice To Have Skills
  • Experience with workflow orchestration tools such as Apache Airflow or Amazon MWAA.
  • Familiarity with dbt for data transformation and analytics engineering.
  • Experience with cloud data platforms such as Snowflake or Databricks.
  • Knowledge of streaming and event-driven data architectures using Apache Kafka, Amazon MSK, or Amazon Kinesis.
  • Experience with Amazon EMR for large-scale distributed data processing.
  • Familiarity with AWS Database Migration Service (DMS) and Change Data Capture (CDC) processes.
  • Experience implementing data ingestion, migration, or synchronization solutions across heterogeneous data sources.
  • Exposure to AI/ML or Generative AI (GenAI) projects, including data preparation, processing pipelines, or integration with machine learning workflows.
  • Experience optimizing cloud data infrastructure for scalability, reliability, and cost efficiency.

Avenue Code reinforces its commitment to privacy and to all the principles guaranteed by the most accurate global data protection laws, such as GDPR, LGPD, CCPA and CPRA. The Candidate data shared with Avenue Code will be kept confidential and will not be transmitted to disinterested third parties, nor will it be used for purposes other than the application for open positions. As a Consultancy company, Avenue Code may share your information with its clients and other Companies from the AI/R Group to which Avenue Code’s consultants are allocated to perform its services.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Data Engineer
Data Engineer

Jobgether • Brazil

Remote
BRL 80,000 - 120,000
Fully remote
Full-time
Talent pool access
+2
Data Engineer
Data Engineer

Sigmasoftware2 • Brazil

Remote
BRL 180,000 - 280,000
Senior Data Engineer (Databricks & AI Solutions)
Senior Data Engineer (Databricks & AI Solutions)

Cognizant Technology Solutions Corporation • São Paulo

Hybrid
BRL 240,000 - 360,000
Data Engineer ID43407
Data Engineer ID43407

AgileEngine, LLC. • Belo Horizonte

Hybrid
BRL 624,000 - 936,000
Professional growth
Competitive compensation
A selection of exciting projects
+1
Lead Data Engineer
Lead Data Engineer

Jobgether • Brazil

On-site
BRL 180,000 - 300,000
Healthcare benefits
Paid time off
Professional certification programs
+1
Data Engineer (Azure) - Remote, Latin America
Data Engineer (Azure) - Remote, Latin America

Bluelight • Belo Horizonte

On-site
BRL 111,600 - 167,400
Senior Data Engineer
Senior Data Engineer

EPAM Systems • Brazil

Remote
BRL 180,000 - 290,000
Healthcare benefits
Paid time off
Upskilling / certifications
+2
Data Engineer (Lead) ID52236
Data Engineer (Lead) ID52236

AgileEngine, LLC. • Curitiba

Hybrid
BRL 624,000 - 936,000
Professional growth: Mentorship and TechTalks
Competitive compensation with budgets for education and fitness
Exciting projects with Fortune 500 companies
+1
Data Engineer (Lead) ID52236
Data Engineer (Lead) ID52236

AgileEngine, LLC. • Brasília

Hybrid
BRL 624,000 - 936,000
Professional growth: Mentorship, TechTalks, and personalized growth roadmaps
Competitive compensation: USD-based pay with education, fitness, and team activity budgets
Exciting projects: Modern solutions with Fortune 500 and top product companies
+1
Data Engineer (Lead) ID52236
Data Engineer (Lead) ID52236

AgileEngine, LLC. • Belo Horizonte

Hybrid
BRL 421,496 - 579,557
Mentorship and personalized growth roadmaps
Competitive compensation with budgets for fitness and education
Exciting projects with Fortune 500 brands
+1