DevOps Engineer - CI/CD for Data Pipelines

Artha Nexgen; Virginia, Northern (IL, KY); Hybrid; USD 120,000 - 180,000; Full time

Job summary

TMCI is seeking a DevOps Engineer to join our team to automate CI/CD pipelines and support data engineering efforts. The role requires experience in standing up DevOps infrastructure, CI/CD automation, and optimizing data workflows within an Agile Scrum team.

The ideal candidate will collaborate with data engineers and software developers to enhance automation, efficiency, and reliability of data pipelines and cloud infrastructure across on-prem and cloud environments.

Qualifications

• 5+ years of experience in DevOps, CI/CD automation, and cloud infrastructure management.
• Experience setting up and managing CI/CD pipelines using GitLab CI/CD or similar tools.
• Experience with integrating automated testing (unit, integration, performance, and security testing) into CI/CD pipelines to ensure software and data pipeline reliability.
• Experience with test automation frameworks (e.g., PyTest, Selenium, JUnit) for CI/CD pipelines.
• Professional experience working with cloud platforms (AWS, Azure, or GCP).
• Experience working with containerization and orchestration tools such as Docker and Kubernetes.
• Strong scripting skills in Python, Bash, or PowerShell for automation and deployment.
• Hands-on experience with monitoring and logging tools (Prometheus, Grafana, ELK, or CloudWatch).
• Familiarity with data pipeline orchestration using tools like Apache Airflow, AWS Step Functions, or Prefect
• Experience with SQL and ETL/ELT data engineering workflows.
• Experience with version control systems (Git, GitHub, GitLab) and branching strategies.
• Strong understanding of networking, security, and IAM in cloud environments.
• Experience with Agile Scrum methodology.

Responsibilities

• Design, build, and manage CI/CD pipelines to support continuous integration, deployment, and automation of data engineering workflows
• Automate deployment processes for data pipelines and applications
• Monitor and optimize system performance, ensuring high availability and fault tolerance
• Support data engineers in building, deploying, and optimizing data pipelines for batch and real-time processing
• Optimize data workflows, ensuring efficient ETL/ELT processes in cloud and on-prem environments.
• Implement monitoring and logging for data pipelines, ensuring reliability and proactive issue resolution
• Collaborate with data engineers to streamline development and deployment processes using DevOps best practices
• Participate as an active member of an Agile Scrum team, contributing to sprint planning, daily standups, and retrospectives
• Work closely with developers, data engineers, and product teams to understand infrastructure and automation needs
• Provide technical guidance on best practices for CI/CD, DevOps, and cloud infrastructure management.
• Contribute to the design, documentation, and continuous improvement of DevOps and data engineering processes

Skills

DevOps

CI/CD automation

GitLab CI/CD

Cloud platforms

Docker

Kubernetes

Python

Bash

PowerShell

Prometheus

Grafana

ELK

CloudWatch

Airflow

SQL

ETL/ELT

Git

IAM

Agile Scrum

Education

Bachelor's degree in CS

Tools

GitLab CI/CD

PyTest

Selenium

JUnit

AWS

Azure

GCP

Job description

TMCI is seeking a DevOps Engineer to join our team to automate CI/CD pipelines and support data engineering efforts. The role requires experience in standing up DevOps infrastructure, CI/CD automation, and optimizing data workflows within an Agile Scrum team.

The ideal candidate will collaborate with data engineers and software developers to enhance automation, efficiency, and reliability of data pipelines and cloud infrastructure across on-prem and cloud environments.