Data Engineer

Agility Partners; Carmel (IN); On-site; USD 120,000 - 160,000; Full time

Job summary

Agility Partners is seeking an Engineer to design and build cloud-based data platforms and backend services for a client in the energy sector. You will create scalable data pipelines using Azure Data Factory, Synapse, and Data Lake, while developing REST APIs with FastAPI and Python.

The role emphasizes data quality, automated testing, and CI/CD, plus integrating AI capabilities and model-serving APIs into the platform.

Qualifications

• Hands-on experience with Azure data services and data pipelines.
• Strong Python development and API integration skills.
• Familiarity with CI/CD, testing, and deployment best practices.
• Knowledge of asynchronous architectures and event-driven systems.

Responsibilities

• Design, build, and maintain scalable data pipelines and orchestration workflows using Azure tools.
• Develop backend services and REST APIs with FastAPI and Python.
• Implement data quality checks and CI/CD processes to improve reliability.
• Integrate ML/AI capabilities into data platforms and applications.

Skills

Python programming

Data engineering

API development

CI/CD practices

Asynchronous/event-driven architecture

Tools

Azure Data Factory

Azure Synapse

Azure Data Lake Storage Gen2

SQL

FastAPI

Azure Functions

Containerization

GitHub

Job description

Agility Partners is seeking a qualified Engineer to fill an open position with one of our clients in the energy industry. This is an exciting opportunity to build and enhance cloud-based data platforms, backend services, and AI-enabled applications that support data-driven business solutions. The ideal candidate will combine data engineering and software development expertise to create reliable data pipelines, scalable APIs, and integrated analytics capabilities using modern Azure technologies.

A Little About This Gig

• Design, develop, and maintain scalable data pipelines and orchestration workflows using Azure Data Factory, Azure Synapse, Azure Data Lake Storage Gen2, SQL, and Python.
• Build and support backend services, REST APIs, and application integrations using Python, FastAPI, Azure Functions, and containerized solutions.
• Implement data quality checks, validation frameworks, automated testing, and CI/CD processes to improve reliability and streamline deployment.
• Integrate machine learning and AI capabilities into data platforms and applications, including feature pipelines, model-serving APIs, and agentic architectures.

The Ideal Candidate

• Hands-on experience with data engineering tools and technologies, including Azure Data Factory, Azure Synapse, Azure Data Lake Storage Gen2, SQL, and Python.
• Experience developing and maintaining data pipelines, orchestration workflows, and data integration solutions.
• Proficiency in Python-based software development, including building backend services and REST APIs with frameworks such as FastAPI.
• Familiarity with Azure Functions, containerized services, and cloud-based application architecture.
• Experience with GitHub, CI/CD pipelines, automated testing, and software deployment best practices.
• Understanding of asynchronous or event-driven architectures and integrating applications with data platforms.