Streamsets Developer

Tata Consultancy Services; Bengaluru; On-site; INR 600,000 - 1,000,000; Full time

Job summary

Join a forward-thinking company as a Data Engineer, where you'll design and implement innovative data pipelines using StreamSets. This role offers the opportunity to work with cutting-edge technologies in cloud environments, ensuring seamless data integration and processing. Collaborate with experts to optimize data flows and enhance pipeline performance, all while contributing to impactful projects that drive business insights. If you're passionate about data engineering and thrive in a dynamic environment, this role is perfect for you!

Qualifications

• 4-10 years of experience in data engineering with a focus on Streamsets.
• Hands-on experience with ETL/ELT pipelines and data integration.

Responsibilities

• Design and develop end-to-end data pipelines using StreamSets.
• Integrate data from diverse sources including cloud storage and APIs.

Skills

Streamsets

Python

SQL

ETL/ELT pipelines

Data integration patterns

Real-time data processing

Stream processing frameworks

Cloud environments

Data Lake/Warehouse

Git

CI/CD pipelines

Tools

StreamSets Data Collector

StreamSets Control Hub

AWS

Azure

GCP

Kafka

Job description

3 days ago Be among the first 25 applicants

Required Skillset: Streamsets, Python, SQL

Experience Range: 4-10 Years

Job Description

Must Have:

1. Hands-on experience with Streamsets (especially with Data Collectors and Control Hub)
2. Strong knowledge of SQL, ETL/ELT pipelines, and data integration patterns
3. Experience with real-time data processing and stream processing frameworks
4. Experience with cloud environments and Data Lake/Warehouse
5. Familiarity with source control tools (e.g., Git) and CI/CD pipelines

Good To Have:

1. Knowledge of Python, Scala, or Java for scripting and automation
2. Understanding of data privacy, compliance, and governance frameworks

Responsibilities / Expectations:

1. Design, develop, and deploy end-to-end data pipelines using StreamSets Data Collector and StreamSets Control Hub
2. Integrate data from diverse sources including relational databases, cloud storage (AWS/Azure/GCP), APIs, and streaming platforms like Kafka
3. Collaborate with data architects and business analysts to understand data requirements and implement solutions accordingly
4. Monitor data flows and troubleshoot issues related to data quality, latency, and pipeline failures
5. Optimize pipeline performance, reliability, and scalability