Senior Data Engineer

IG Soft; Anupgarh; On-site; INR 1,500,000 - 2,100,000; Full time

Job summary

IG Soft is seeking a Senior Data Engineer to join our agile team and help build scalable data pipelines for both batch and streaming data. You will automate ingestion, perform transformations, and load data into both relational and NoSQL stores.

You will provide data gateways for querying stored data and work closely with data science to enable data-driven decision making. The ideal candidate will have a strong background in distributed systems and a willingness to learn new technologies.

Qualifications

• 5+ years of experience as a Data Engineer.
• Experience with relational and NoSQL databases and data modeling.
• Proficient in building ETL/ELT pipelines and data ingestion from multiple sources.

Responsibilities

• Implement, maintain, and evolve an optimal data pipeline architecture across batch and streaming data.
• Automate data ingestion and transformations; load data into relational and NoSQL stores.
• Create unified data gateways for querying stored data and support data science teams.
• Collaborate with teams to customize analytics tools and enable rapid prediction modelling cycles.
• Explore and apply modern Big Data technologies to enhance the data pipeline.

Skills

SQL

Data modeling

NoSQL

REST APIs

GraphQL

gRPC

Python

Go

Distributed systems

Data pipelines

English fluency

Education

Graduate degree in Software Engineering or similar

Tools

Hadoop ecosystem

Apache Airflow

Apache Parquet

Apache Avro

Apache Beam

Apache Impala

Apache Kudu

ScyllaDB

RabbitMQ

Job description

Who We Are

IGSoft LTD is a European software development company with a wide range of international clients.

For more than 15 years we have been developing and maintaining one of the leading and easiest to use online gaming platforms in the world. Tens of thousands of players trust us every day and we have hundreds of corporate clients.

We have offices based in Sofia and Belgrade, with more than 50 staff members. As our business grows, we now seek to expand our family with bright and talented people who are passionate about the software industry.

About The Role

For the future growth of company, we are looking for a motivated Senior Data Engineer to come join our agile team of professionals and be part of creating complex solutions:

As a data engineer you will be responsible for building and optimizing an effective data pipeline architecture encompassing both batch and streaming data as well as for maintaining/evolving a large-scale data system. Your main duties will revolve around automating ingestion of data from various external and internal data sources, doing necessary transformations, and loading data into different data stores (both relational and NoSQL).

Furthermore, you will provide required data gateways for querying the stored data in a unified and versatile manner. As a date engineer you will work tightly with a data science team enabling them to do their jobs more effectively in a data-driven fashion.

The ideal candidate is an experienced automation engineer capable of building distributed solutions from the ground up considering the needs of multiple teams. We are seeking for a self-directed tech-savvy person who is willing to learn new technologies and use them to solve data science problems by implementing a consistent data delivery architecture.

Responsibilities for Data Engineer

• Implement, maintain, and evolve an optimal data pipeline architecture. Perform data acquisition from multitude of external and internal sources using different communication protocols.
• Write scripts/services to automate tasks at every stage of a data pipeline.
• Integrate disparate infrastructure components into a unified data platform that will support other teams in their work.
• Customize analytics tools that utilize the data pipeline to provide a platform capable of supporting rapid prediction modelling cycles.
• Propose innovative approaches to improve the existing data pipeline using modern Big Data technologies.

Qualifications For Data Engineer

• Excellent SQL and database design knowledge, and experience working with both relational and NoSQL databases.
• Experience in handling unstructured, semi-structured, and structured data sets.
• Experience in crafting Extract-Transform-Load jobs to wrangle data into proper target formats.
• Experience in interacting with remote systems using REST APIs, GraphQL endpoints, gRPC services, message queues, and Big Data file and object stores.
• 5+ years of experience as a Data Engineer with a Graduate degree in Software Engineering or another similar field. A candidate should preferably have experience using the following technologies (or equivalent ones):
• Experience with the core Hadoop ecosystem
• Experience with columnar relational and NoSQL databases (like, Apache Kudu and ScyllaDB)
• Experience with workflow management tools (like, Apache Airflow) Experience with Big Data data formats (like, Apache Parquet, Apache Avro, etc.)
• Experience with stream processing systems (like, Apache Beam) Experience with mainstream languages (like, Python, Go, etc.) Experience with query engines (like, Apache Impala)
• Experience with RabbitMQ or similar messaging hub Good communication skills and fluency in English.

Your voice and opinion are important to us. We will support your personal growth and give you the opportunity to learn new technologies. We do work in dynamic environment, but we also know how to praise the good work.