Senior Data Engineer

Kiana Ai; Kochi; On-site; INR 800,000 - 1,200,000; Full time

Job summary

A leading generative AI consulting firm is seeking a Senior Data Engineer to design and maintain data pipelines, optimize storage solutions, and lead data integration projects. The ideal candidate will have over 5 years of experience with Azure Cloud services and strong programming skills. This role offers an opportunity to work with data scientists and engineers on high-impact projects. The position is based in Kochi, Kerala.

Qualifications

• 5+ years of experience in data engineering, focusing on cloud-based solutions.
• Proficiency in Azure Cloud services and strong programming skills in Python and SQL.
• Experience with CI/CD pipelines and version control systems.

Responsibilities

• Design, develop, and maintain robust data pipelines and ETL processes.
• Collaborate with stakeholders to understand data requirements and deliver solutions.
• Monitor and troubleshoot data pipeline performance.

Skills

Azure Data Factory

Databricks

Python

SQL

Data Lake

Cosmos DB

Data Warehousing

ETL tools

CI/CD pipelines

Education

Bachelor's degree in Computer Science, Engineering, or related field

Master's degree preferred

Tools

Docker

Kubernetes

Tableau

Power BI

Job description

As a Senior Data Engineer at Kiana AI, you will be responsible for designing, developing, and maintaining robust data pipelines and architectures that support our AI and machine learning initiatives. You will work closely with data scientists, software engineers, and other stakeholders to ensure the availability, reliability, and scalability of our data infrastructure.

Requirements

Key Responsibilities:

Design, develop, and maintain robust data pipelines and ETL processes using Azure Data Factory, Databricks, and other Azure services.

Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver high-quality data solutions.

Optimize and maintain data storage solutions, including Azure Data Lake, SQL Database, and Cosmos DB.

Implement and manage data integration solutions to ensure data consistency, accuracy, and availability.

Monitor and troubleshoot data pipeline performance, ensuring reliability and scalability.

Develop and enforce data governance and security standards to protect sensitive data.

Mentor junior data engineers and provide technical leadership within the team.

Qualifications:

Bachelor's degree in Computer Science, Engineering, or a related field (Master's degree preferred).

5+ years of experience in data engineering, with a focus on cloud-based solutions.

Proficiency in Azure Cloud services, including Azure Data Factory, Databricks, Data Lake, SQL Database, and Cosmos DB.

Strong programming skills in Python, SQL, and experience with ETL tools.

Familiarity with data modeling, data warehousing, and big data technologies.

Experience with CI/CD pipelines and version control systems (e.g., Git).

Excellent problem-solving skills and attention to detail.

Strong communication and collaboration skills.

Preferred Skills:

Experience with data orchestration tools (e.g., Apache Airflow, Luigi).

Knowledge of machine learning and AI frameworks.

Familiarity with containerization and orchestration tools (e.g., Docker, Kubernetes).

Experience with data visualization tools (e.g., Tableau, Power BI).

Experience with machine learning and data science frameworks.

About the Company

Kiana AI is a leading generative AI consulting firm dedicated to helping businesses harness the power of artificial intelligence to drive innovation and success. We are seeking a highly skilled Senior Data Engineer to join our team and contribute to our advanced data projects.