Senior Data Engineer – Kafka & Flink

Blackstraw AI; Chennai District; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

A technology services company in Chennai seeks a Senior Data Engineer to design and develop scalable data pipelines using Apache Kafka and Flink. The ideal candidate will have 5-8 years of experience and expertise in AWS services and MLOps practices. This role involves collaborating with cross-functional teams to enhance data workflows and processing solutions, ensuring high-performance ETL and real-time pipelines. Join a dynamic team focused on delivering innovative AI solutions.

Qualifications

• Extensive experience in building data pipelines and streaming applications using Apache Kafka and Apache Flink.
• Proficient in AWS services including SageMaker, S3, Glue, Lambda, and CloudFormation/Terraform.
• Hands‑on expertise with MLOps best practices, including model versioning, monitoring, and CI/CD for ML pipelines.

Responsibilities

• Design, develop, and maintain high‑performance ETL and real‑time data pipelines using Apache Kafka and Apache Flink.
• Build scalable and automated MLOps pipelines for ML model training and deployment.
• Collaborate with data scientists and DevOps teams to streamline model deployment.

Skills

Apache Kafka

Apache Flink

AWS SageMaker

Terraform

Python

SQL

CI/CD

MLOps best practices

Education

Bachelor’s degree in Computer Science or related field

Tools

AWS S3

AWS Lambda

AWS Glue

CloudFormation

Job description

Senior Data Engineer – Kafka & Flink

5 to 8 Years Experience | India | All Jobs

Company name: Blackstraw.ai

Office location: Chennai

Job Type: Full-time

Experience: 5 to 7 years

Job Description: The candidate should have extensive experience in designing and developing scalable data pipelines and real‑time data processing solutions. As a key member of the team, the Senior Data Engineer will play a critical role in building end‑to‑end data workflows, supporting machine learning model deployment, and driving MLOps practices in a fast‑paced, agile environment.

Strong expertise in Apache Kafka, Apache Flink, AWS SageMaker, and Terraform is essential. Additional experience with infrastructure automation and CI/CD for ML models is a significant advantage.

Key Responsibilities:

• Design, develop, and maintain high‑performance ETL and real‑time data pipelines using Apache Kafka and Apache Flink.
• Build scalable and automated MLOps pipelines for training, validation, and deployment of models using AWS SageMaker and associated services.
• Implement and manage Infrastructure as Code (IaC) using Terraform to provision and manage AWS environments.
• Collaborate with data scientists, ML engineers, and DevOps teams to streamline model deployment workflows and ensure reliable production delivery.
• Optimize data storage and retrieval strategies for large‑scale structured and unstructured datasets.
• Develop data transformation logic and integrate data from various internal and external sources into data lakes and warehouses.
• Monitor, troubleshoot, and enhance performance of data systems in a cloud‑native, fast‑evolving production setup.
• Ensure adherence to data governance, privacy, and security standards across all data handling activities.
• Document data engineering solutions and workflows to facilitate cross‑functional understanding and ongoing maintenance.

Required Qualifications:

• Bachelor’s degree in Computer Science or related field.
• Extensive experience in building data pipelines and streaming applications using Apache Kafka and Apache Flink.
• Strong experience in ETL development, data modeling, and managing data in large‑scale environments.
• Proficient in AWS services including SageMaker, S3, Glue, Lambda, and CloudFormation/Terraform.
• Hands‑on expertise with MLOps best practices, including model versioning, monitoring, and CI/CD for ML pipelines.
• Proficiency in Python and SQL; experience with Java is a plus for streaming jobs.
• Deep understanding of cloud infrastructure automation using Terraform or similar IaC tools.
• Excellent problem‑solving skills with the ability to troubleshoot data processing and deployment issues.
• Experience in fast‑paced, agile development environments with frequent delivery cycles.
• Strong communication and collaboration skills to work effectively across cross‑functional teams.

Company Profile:

Blackstraw.ai is an end‑to‑end technology services company specializing in Artificial Intelligence (AI) and Engineering solutions across Data Science, Data Engineering, LLM/GenAI and LLMOps.

Founded in 2018, we help global enterprises across North America, Europe and Asia to build and operationalize AI systems that create measurable business impact. Our mission is to make AI adoption simpler, faster and scalable through a blend of deep domain expertise, reusable accelerators and proven engineering practices.

With a 450+ strong team of engineers, data scientists and AI specialists, we partner with organizations to deliver real‑world outcomes in areas such as predictive analytics, computer vision, natural language processing and Generative AI.

At Blackstraw.ai, we’re passionate about solving complex business challenges through intelligent automation, modern data architectures and next‑gen AI applications – enabling enterprises to move from AI experimentation to transformation.

Headquartered in Florida (USA) with operations in Canada and India, Blackstraw.ai continues to empower global enterprises to unlock the true potential of AI.