Data Engineer

IBS Software; Thiruvananthapuram; On-site; INR 1,500,000 - 2,100,000; Full time

Job summary

IBS Software in India is seeking a data engineer to design, build and maintain data processing ETL pipelines in a multi-cloud, multi-region environment, delivering scalable data products.

You will work on data architecture, streaming data, warehousing with Snowflake, and using Docker/Kubernetes for deployment, collaborating with cross-functional teams to resolve technical and operational issues.

Qualifications

• Experience designing data pipelines and ETL/ELT architectures.
• Understanding of data mesh or data fabric concepts.
• Proficiency with Python, Java or Scala in data engineering.
• Experience with Snowflake or data warehousing solutions.
• Familiarity with streaming data technologies.
• Version control with Git.
• Knowledge of containerization (Docker) and orchestration (Kubernetes).

Responsibilities

• Design, build and maintain ETL pipelines and data products.
• Drive data investigations to resolve technical and operational issues.
• Solve complex business problems with scalable, robust data solutions using appropriate technologies.

Skills

Data engineering patterns

Data architecture

Big Data Technologies

Programming Languages

Data Warehousing

ETL/ELT Tools

Database Systems

Data Modelling

Streaming Data

Version Control

Containerization and Orchestration

Data Security

Monitoring and Logging

Tools

Docker

Kubernetes

Git

Snowflake

EMR

Glue

Orchestration tools

Job description

Job Description:

What you’ll do

• Design, build and maintain data processing ETL pipelines & data products in a multi-cloud, multi-region, distributed processing context choosing the right technologies.
• Drive data investigations to deliver a resolution of technical, procedural, and operational issues.
• Solve complex business problems by utilizing disciplined development methodology, producing scalable, flexible, efficient and supportable solutions using appropriate technologies.

Skills

• Data engineering patterns: Sound knowledge of the different data engineering patterns to determine what to use and when
• Data architecture: Understanding of data architecture and frameworks like Data mesh, data fabric etc.
• Big Data Technologies: Proficiency in big data technologies such as data lake, EMR, Glue for analysing large datasets.
• Programming Languages: Proficiency in programming languages commonly used in data engineering, such as Python, Java, or Scala. with OOP expertise
• Data Warehousing: Strong knowledge of data warehousing solutions preferably Amazon Snowflake.
• ETL/ELT Tools: Familiarity with orchestration tools
• Database Systems: Expertise in both relational and NoSQL databases.
• Data Modelling: Skill in designing efficient data models, both for OLAP and OLTP systems.
• Streaming Data: Knowledge of streaming data technologies.
• Version Control: Experience with version control systems : Git.
• Containerization and Orchestration: Understanding of containerization technologies (Docker) and container orchestration platforms (Kubernetes).
• Data Security: Knowledge of data encryption, access control, and compliance with data privacy regulations.
• Monitoring and Logging: Proficiency in setting up monitoring and logging solutions for data pipelines using different tools