Data Engineer

Prediktive; Atlanta (GA); Remote; USD 120,000 - 180,000; Full time

Job summary

Prediktive is seeking an experienced Data Engineer to design and maintain scalable data pipelines and data warehousing solutions. You will optimize ETL workflows, work with Azure Data Factory and Azure Synapse, and ensure data integrity and security in a regulated setting.

You will collaborate with cross-functional teams on real-time and batch data processing for AI-enabled applications, leveraging Docker and Kubernetes for containerized services.

Qualifications

• Advanced level of English.
• 6+ years of experience with Python for data manipulation (Pandas, PySpark) and building custom API integrations.
• Hands-on experience with ETL processes and tools, including pipeline design and optimization.
• Experience with orchestration frameworks (Airflow, Prefect, or similar) and complex data/ML pipelines.
• Strong understanding of security protocols (OAuth, Key Vault, Private Links) and data governance in regulated environments.
• Professional experience with Git and Azure DevOps/GitHub Actions for automated testing and deployment of data code.
• Strong experience with both relational (PostgreSQL, SQL Server) and NoSQL databases.
• Familiarity with Docker and Kubernetes for deploying containerized data services and microservices.

Responsibilities

• Design, build, and maintain scalable data pipelines using Python for ingestion, transformation, and processing.
• Develop and optimize ETL workflows to support analytics and AI-driven applications.
• Architect and manage data warehousing solutions with high performance.
• Work with Azure data services (Azure Data Factory, Azure Synapse) to implement robust data solutions.
• Ensure data integrity, security, and compliance with bank-grade infrastructure standards.
• Monitor and troubleshoot data pipelines and resolve performance bottlenecks.
• Implement best practices for data governance, documentation, and version control.
• Support real-time and batch data processing for AI applications.

Skills

Python

Pandas

PySpark

ETL pipelines

Airflow

OAuth/Key Vault

Git/GitHub Actions

Azure

SQL

NoSQL

Docker

Kubernetes

Education

Bachelor's degree

Tools

Azure Data Factory

Azure Synapse Analytics

PostgreSQL

SQL Server

NoSQL databases

Docker

Kubernetes

Git

Azure DevOps

GitHub Actions

Job description

We are looking for a Data Engineer based in Latin America to work on a long-term project for one of our clients, a Software Development company based in Atlanta, Georgia.Our client is building a secure, bank-grade AI infrastructure and developing cutting-edge applications designed to power the next generation of modern banking and financial services.

Responsibilities

• Design, build, and maintain scalable data pipelines using Python for data ingestion, transformation, and processing.
• Develop and optimize ETL workflows to support analytics and AI-driven applications.
• Architect and manage data warehousing solutions, ensuring high performance and reliability.
• Work with Azure data services (e.g., Azure Data Factory, Azure Synapse Analytics) to implement robust data solutions.
• Ensure data integrity, security, and compliance aligned with bank-grade infrastructure standards.
• Monitor and troubleshoot data pipelines, identifying and resolving performance bottlenecks.
• Implement best practices for data governance, documentation, and version control.
• Support real-time and batch data processing needs for cutting-edge AI applications.

Requirements

• Advanced Level of English.
• 6+ years of experience working with Python, specifically for data manipulation (Pandas, PySpark) and building custom API integrations.
• Hands-on experience with ETL processes and tools, including pipeline design and optimization.
• Experience with orchestration frameworks (e.g., Airflow, Prefect, or similar) and complex data/ML pipelines.
• Strong understanding of security protocols (OAuth, Key Vault, Private Links) and data governance frameworks within a regulated environment.
• Professional experience with Git and Azure DevOps/GitHub Actions for automated testing and deployment of data code.
• Strong experience with both relational (PostgreSQL, SQL Server) and NoSQL databases.
• Familiarity with Docker and Kubernetes for deploying containerized data services and microservices.

Bonus Points

• Bachelor’s Degree in Computer Science, Systems Engineering or related fields.

What we Offer

• Long term positions
• Compensation in USD
• Cool clients and products
• Work with great engineers