Devops Engineer

Mothership

San Antonio (TX)

On-site

USD 100,000 - 150,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

A leading company in the tech industry is seeking an experienced Data Engineer to design and implement scalable data processing pipelines using Apache Flink and Kafka on AWS. The ideal candidate will possess strong skills in AWS services, automation, and real-time data processing while collaborating with various teams to meet performance and security standards.

Qualifications

  • Proficiency in AWS services (Amazon MSK, Kinesis, Lambda, etc.).
  • Experience with Apache Flink and Kafka management.
  • Strong automation skills with Terraform or CloudFormation.

Responsibilities

  • Design and deploy scalable data processing pipelines using Apache Flink and Kafka.
  • Manage and optimize Kafka and Flink clusters on AWS.
  • Implement security mechanisms for Kafka and Flink deployments.

Skills

AWS services
Infrastructure as Code
Real-time stream processing
CI/CD pipelines
Docker
Kubernetes
Scripting in Python
Monitoring tools

Job description

  • Proficiency in AWS services such as Amazon MSK (Managed Streaming for Kafka), Amazon Kinesis, AWS Lambda, Amazon S3, Amazon EC2, Amazon RDS, Amazon VPC, and AWS IAM.
  • Ability to manage infrastructure as code with AWS CloudFormation or Terraform.
  • Understanding of Apache Flink for real-time stream processing and batch data processing.
  • Familiarity with Flinks integration with Kafka, or other messaging services.
  • Experience in managing Flink clusters on AWS (using EC2, EKS, or managed services).
  • Deep knowledge of Kafka architecture, including brokers, topics, partitions, producers, consumers, and zookeeper.
  • Proficiency with Kafka management, monitoring, scaling, and optimization.
  • Hands-on experience with Amazon MSK (Managed Streaming for Kafka) or self-managed Kafka clusters on EC2.
  • DevOps & Automation:
  • Strong experience in automating deployments and infrastructure provisioning.
  • Familiarity with CI/CD pipelines using tools like Jenkins, GitLab, GitHub Actions, CircleCI, etc.
  • Experience with Docker and Kubernetes, especially for containerizing and orchestrating applications in cloud environments.
  • Programming & Scripting:
  • Strong scripting skills in Python, Bash, or Go for automation tasks.
  • Ability to write and maintain code for integrating data pipelines with Kafka, Flink, and other data sources.
  • Monitoring & Performance Tuning:
  • Knowledge of CloudWatch, Prometheus, Grafana, or similar monitoring tools to observe Kafka, Flink, and AWS service health.
  • Expertise in optimizing real-time data pipelines for scalability, fault tolerance, and performance.

Responsibilities:

  • Infrastructure Design & Implementation:
  • Design and deploy scalable and fault-tolerant real-time data processing pipelines using Apache Flink and Kafka on AWS.
  • Build highly available, resilient infrastructure for data streaming, including Kafka brokers and Flink clusters.
  • Platform Management:
  • Manage and optimize the performance and scaling of Kafka clusters (using MSK or self-managed).
  • Configure, monitor, and troubleshoot Flink jobs on AWS infrastructure.
  • Oversee the deployment of data processing workloads, ensuring low-latency, high-throughput processing.
  • Automation & CI/CD:
  • Automate infrastructure provisioning, deployment, and monitoring using Terraform, CloudFormation, or other tools.
  • Integrate new applications and services into CI/CD pipelines for real-time processing.
  • Collaboration with Data Engineering Teams:
  • Work closely with Data Engineers, Data Scientists, and DevOps teams to ensure smooth integration of data systems and services.
  • Ensure the data platforms scalability and performance meet the needs of real-time applications.
  • Security and Compliance:
  • Implement proper security mechanisms for Kafka and Flink clusters (e.g., encryption, access control, VPC configurations).
  • Ensure compliance with organizational and regulatory standards, such as GDPR or HIPAA, where necessary.
  • Optimization & Troubleshooting:
  • Optimize Kafka and Flink deployments for performance, latency, and resource utilization.
  • Troubleshoot issues related to Kafka message delivery, Flink job failures, or AWS service outages.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Confluent Kafka & Flink Streaming Architect
Senior Confluent Kafka & Flink Streaming Architect

TechDigital Group • Columbus (OH)

On-site
USD 180,000 - 280,000
Senior Network Engineer - Versa & Cloud Networking Expert
Senior Network Engineer - Versa & Cloud Networking Expert

TechDigital Group • Massachusetts

On-site
USD 70,000 - 110,000
Kafka / Flink Engineer - Onsite
Kafka / Flink Engineer - Onsite

Global Business Ser. 4u Inc. • New York (NY)

On-site
USD 120,000 - 150,000
Software Engineer: IV (Lead)
Software Engineer: IV (Lead)

Dale Workforce Solutions • Saint Paul (MN)

On-site
USD 120,000 - 160,000
DevOps Lead
DevOps Lead

TechDigital Group • San Antonio (TX)

On-site
USD 120,000 - 180,000
Senior Kafka, Confluent DevOps Administrator with Terraform
Senior Kafka, Confluent DevOps Administrator with Terraform

Advanced Image Inc. • St. Louis (MO)

On-site
USD 120,000 - 180,000
Senior Kafka Engineer / Administrator Hybrid
Senior Kafka Engineer / Administrator Hybrid

Generis Tek Inc • Denver (CO)

Hybrid
USD 55,000 - 62,000
Kafka Engineer / Administrator
Kafka Engineer / Administrator

Salesforce, Inc. • Denver (CO)

On-site
USD 140,000 - 180,000
Senior Data Streaming Engineer
Senior Data Streaming Engineer

First Soft Solutions LLC • Woodlawn (MD)

On-site
USD 140,000 - 190,000
Onsite work 5 days/week
Senior Kafka Platform Engineer
Senior Kafka Platform Engineer

Selby Jennings • Chicago (IL)

On-site
USD 120,000 - 150,000