Senior Data Engineer - Azure Data & AI Pipelines (LATAM)

Prediktive; Atlanta (GA); Remote; USD 120,000 - 180,000; Full time

Job summary

Prediktive is seeking an experienced Data Engineer to design and maintain scalable data pipelines and data warehousing solutions. You will optimize ETL workflows, work with Azure Data Factory and Azure Synapse, and ensure data integrity and security in a regulated setting.

You will collaborate with cross-functional teams on real-time and batch data processing for AI-enabled applications, leveraging Docker and Kubernetes for containerized services.

Qualifications

• Advanced level of English.
• 6+ years of experience with Python for data manipulation (Pandas, PySpark) and building custom API integrations.
• Hands-on experience with ETL processes and tools, including pipeline design and optimization.
• Experience with orchestration frameworks (Airflow, Prefect, or similar) and complex data/ML pipelines.
• Strong understanding of security protocols (OAuth, Key Vault, Private Links) and data governance in regulated environments.
• Professional experience with Git and Azure DevOps/GitHub Actions for automated testing and deployment of data code.
• Strong experience with both relational (PostgreSQL, SQL Server) and NoSQL databases.
• Familiarity with Docker and Kubernetes for deploying containerized data services and microservices.

Responsibilities

• Design, build, and maintain scalable data pipelines using Python for ingestion, transformation, and processing.
• Develop and optimize ETL workflows to support analytics and AI-driven applications.
• Architect and manage data warehousing solutions with high performance.
• Work with Azure data services (Azure Data Factory, Azure Synapse) to implement robust data solutions.
• Ensure data integrity, security, and compliance with bank-grade infrastructure standards.
• Monitor and troubleshoot data pipelines and resolve performance bottlenecks.
• Implement best practices for data governance, documentation, and version control.
• Support real-time and batch data processing for AI applications.

Skills

Python

Pandas

PySpark

ETL pipelines

Airflow

OAuth/Key Vault

Git/GitHub Actions

Azure

SQL

NoSQL

Docker

Kubernetes

Education

Bachelor's degree

Tools

Azure Data Factory

Azure Synapse Analytics

PostgreSQL

SQL Server

NoSQL databases

Docker

Kubernetes

Git

Azure DevOps

GitHub Actions

Job description

Prediktive is seeking an experienced Data Engineer to design and maintain scalable data pipelines and data warehousing solutions. You will optimize ETL workflows, work with Azure Data Factory and Azure Synapse, and ensure data integrity and security in a regulated setting.

You will collaborate with cross-functional teams on real-time and batch data processing for AI-enabled applications, leveraging Docker and Kubernetes for containerized services.