Senior Data Management Specialist 5+Yrs

Rangam Infotech Pvt. Ltd.; Karnataka; On-site; INR 10,00,000 - 15,00,000; Full time

Job summary

A leading tech company in Karnataka is looking for a Data Engineer to build and manage scalable data pipelines. The role involves developing ETL processes, implementing data quality checks, and utilizing AWS services like Glue and Redshift. Ideal candidates will have 5+ years of experience in data engineering, strong programming skills in Python or Scala, and familiarity with big data technologies such as Flink and Spark.

Qualifications

• 5+ years of experience in developing scalable and fault-tolerant data pipelines.
• Proficiency in at least one programming language: Python or Scala.
• Strong understanding of big data technologies: Flink, Spark, Kafka.

Responsibilities

• Develop and maintain scalable ETL pipelines.
• Implement and monitor data quality checks.
• Build solutions using AWS services for data processing.

Skills

ETL pipeline development

Python

Data governance

Data quality checks

Big data technologies

Education

Degree in Computer Science or related field

Tools

AWS Glue

AWS Redshift

AWS EMR

Job description

As the Data engineer in the Lending Data Engineering team, you will work closely with data modelers, product analytics, product managers, software engineers and business stakeholders across the SEA in understanding the business and data requirements. You will be responsible for building and managing the data asset, including acquisition, storage, processing and consumption channels, and using some of the most scalable and resilient open source big data technologies like Flink, Airflow, Spark, Kafka, Trino and more on cloud infrastructure. You are encouraged to think out of the box and have fun exploring the latest patterns and designs.

The Day-to-Day Activities

• Developing and maintaining scalable and reliable ETL pipelines and processes to ingest data from a large number and variety of data sources
• Developing a deep understanding of real-time data productions availability to inform on the real time metric definitions
• Develop data quality checks and establish best practices for data governance, quality assurance, data cleansing, and ETL-related activities
• Build solutions leveraging AWS services such as Glue, Redshift, Athena, Lambda, S3, Step Functions, EMR, and Kinesis to enable efficient data processing and analytics.
• Implement and monitor data quality checks and establish best practices for data governance, quality assurance, data cleansing, and ETL-related activities using AWS Glue DataBrew or similar tools.

The Must-Haves

• At least 5+ years of relevant experience in developing scalable, secured, distributed, fault tolerant, resilient & mission-critical data pipelines.
• Proficiency in at least one of the programming languages Python, Scala.
• Strong understanding of big data technologies like Flink, Spark, Trino, Airflow, Kafka, and familiarity with AWS services like EMR, Glue, Redshift, Kinesis, and Athena.
• Experience with SQL, schema design and data modeling((must have experience in all these skills)
• Experience with different databases – NoSQL, Columnar, Relational.
• Develop familiarity with in-house and AWS-native data platform tools to efficiently set up data pipelines.
• You are organized, insightful and can communicate your observations well, both written and verbally to your stakeholders to share updates and coordinate the development of data pipelines

The Nice-to-Haves

• You have a degree or higher in Computer Science, Electronics or Electrical Engineering, Software Engineering, Information Technology or other related technical disciplines.
• You have a good understanding of Data Structure or Algorithms or Machine Learning models.