Threat Detection Data Engineer: Automation & Cloud Pipelines

Jobtailor; New York (NY); On-site; USD 110,000 - 170,000; Full time

Job summary

Jobtailor seeks an experienced data engineer to design, develop, and maintain scalable data pipelines and analytics products supporting reporting and analytics initiatives. The role emphasizes data quality, governance, and automation across cloud-based architectures and security tooling.

You will collaborate with data scientists, analysts, and engineers to deliver solutions, mentor junior teammates, and ensure reliable data delivery and integration across platforms.

Qualifications

• 5+ years of professional development experience using SQL and Python.
• 5+ years of professional experience building data pipelines (batch and/or streaming) and data governance practices.
• Hands-on experience with modern data warehouses and large-scale data platforms (Databricks Preferred).
• Deep understanding of database structures principles and best practices.
• Deep understanding of data modeling, ETL/ELT design, data engineering best practices, including data integration patterns, replication, event streaming, and pipeline performance tuning.
• Hands-on experience with security automation and orchestration tools and/or software development technologies.
• Comfortable with REST API best practices and usage.
• Hands-on experience deploying content and/or systems through CI/CD environments, and tooling such as GitLab, GitLab CI, Terraform, Ansible, Kubernetes etc.
• Hands-on experience with AWS and cloud services configuration and development.
• Excellent written and verbal communications skills.
• Ability to communicate with technical and non-technical audiences.

Responsibilities

• Design, develop, and maintain scalable data pipelines, datasets, and analytics products supporting the IRC department’s reporting and analytics capabilities.
• Develop and maintain enterprise-ready ETL/ELT processes and data transformation frameworks to support data-driven initiatives.
• Design, build and support cloud-based data pipelines and data lake architectures using software and data engineering best practices.
• Ensure data quality, lineage, integrity, and governance standards are implemented throughout the data lifecycle.
• Optimize and troubleshoot existing data pipelines for performance and reliability, ensuring data quality and integrity across various data sources.
• Identify opportunities to automate workflows/processes and improve efficiency across cybersecurity teams through SOAR playbooks and Kubernetes-hosted and/or cloud-hosted pipelines.
• Assist with the design, engineering, deployment, and maintenance of custom automation tools (primarily in Python and PowerShell).
• Develop integrations with security tools and APIs.
• Evaluate and implement emerging technologies to improve cybersecurity and analytics capabilities.
• Assist with the development and support of dashboards, metrics, and monitoring alerts related to the team’s and the department’s operations.
• Develop and maintain documentation related to the team’s automated workflows/products.
• Partner with data scientists, analysts, engineers, and other stakeholders to identify requirements and deliver high-quality solutions and insights.
• Provide mentorship and coaching to junior team members, fostering skill development and encouraging continuous learning.

Skills

SQL Development

Python Programming

Data Pipeline Engineering

ETL/ELT Design

Security Automation

Tools

Databricks

GitLab

Terraform

Ansible

Kubernetes

AWS Cloud Services

Job description

Jobtailor seeks an experienced data engineer to design, develop, and maintain scalable data pipelines and analytics products supporting reporting and analytics initiatives. The role emphasizes data quality, governance, and automation across cloud-based architectures and security tooling.

You will collaborate with data scientists, analysts, and engineers to deliver solutions, mentor junior teammates, and ensure reliable data delivery and integration across platforms.