Sr.Data Engineer/Scientist

Cloud Counselage Pvt Ltd; Hyderabad; On-site; INR 1,000,000 - 1,500,000; Full time

Job summary

A technology consulting firm is seeking a Senior Data Engineer/Scientist in Hyderabad. The ideal candidate will have 5-8 years of experience, particularly in Python, ETL tools, and data pipelining. Responsibilities include collaborating with teams to develop data systems and optimizing data pipelines. The role requires excellent communication skills and a Bachelor's degree in IT or a related field. Compensation will be according to industry standards and experience.

Qualifications

• 5-8 years of experience in data processing and scripting.
• Hands-on experience with data pipelining and ETL tools.
• Excellent written and verbal communication skills.

Responsibilities

• Collaborate with an agile team to design and develop big data systems.
• Support custom solutions offered to product development.
• Optimize existing data pipelines for speed and reliability.

Skills

Python

ETL tools

AWS Athena

SQL

NoSQL databases

Data pipelining

Kafka

Linux

PySpark

REST APIs

Education

Bachelor's degree in IT or related field

Tools

Hadoop

BigQuery

RedShift

Airflow

Job description

About the job Sr.Data Engineer/Scientist

Looking for Python Ninjas having 5- 8years of experience, to showcase their expertise in a stimulating environment, geared towards building cutting-edgeproducts and services. They should have a knack for data processing, scripting and should be excited about delivering a scalable, high-quality component-based model development framework using big data. They should be Passionate,curious and innovative to think beyond the ordinary.

They are someone whocan start immediately or have 30 days notice period.

Job Responsibilities:

• Collaborating across an agile team to continuously design, iterate, and develop big data systems.
• Working with theirengineering team to support custom solutions offered to the product development.
• Extracting, transforming, and loading data into internal databases and Hadoop
• Understanding their data sets and how to bring them together
• Filling the gap between development, engineering and data ops
• Optimizing our new and existing data pipelines for speed and reliability
• Creating, maintaining and documenting scripts to support ongoing custom solutions
• Deploying new products and product improvements.
• Understanding of best practices, common coding patterns and good practices around
• Storing, partitioning, warehousing and indexing of data
• Documenting and managing multiple repositories of code

Mandatory Requirements:

• Hands-on experience in data pipelining and ETL (Any one tools: Hadoop, BigQuery, RedShift, Athena) and in AirFlowFamiliar with pulling and pushing files from SFTP and AWS S3
• Familiarity with AWS Athena and Redshift is mandatory
• Familiarity with SQL programming to query and transform data from relationalDatabases
• Experience in reading the data from Kafka topic (both live stream and offline)
• Experience with any Cloud solutions including GCP / AWS / OCI / Azure
• Familiarities with Linux (and Linux work environment).
• Excellent written and verbal communication skills.
• Experience in PySpark and Data frames
• Experience with SQL and NoSQL databases (MySQL, Cassandra)

Preferred Requirements:

• Know your way around REST APIs.[Able to integrate not necessary to publish.
• From a product experience background

Qualities:

• Excellent organizational skills, including attention to precise details.
• Strong multitasking skills and ability to work in a fast-paced environment
• 5 years experience in database systems.
• 5+years experience with Python to develop scripts.
• Bachelors in IT or related field

What's for the Candidate

• Compensation as per industry standards or based on the experience and last CTC

Note: For our vision, our Flagship Event' IAC VISION 2030 ' would provide employment and career opportunities for millions of job-ready interns, freshers and professionals in our Industry Academia Community (IAC).

By submitting your application you confirm that you are a member of IAC or give your consent to add you to the IAC platform as a member of Industry Academia Community.