Data Engineer

techcarrot; Hyderabad; On-site; INR 1,400,000 - 2,200,000; Full time

Job summary

techcarrot is seeking an experienced Data Engineer to review and design robust data pipelines and prototypes in a collaborative, cross-functional environment. You will work with Azure Databricks, Spark, and SQL to transform structured and unstructured data into scalable solutions.

The role requires 5-7 years of hands-on experience, a strong foundation in databases, and expertise across the Azure data lake, data factory, and related services.

Qualifications

• Minimum 5-7 years of practical experience as Data Engineer.
• Bachelor’s degree in computer science, software engineering, information technology, or a related field.
• Azure cloud stack in-production experience.

Responsibilities

• You design, architect, implement and test rapid prototypes that demonstrate value of the data and present them to diverse audiences.
• You participate in early state design and feature definition activities.
• Responsible for implementing robust data pipeline using Microsoft, Databricks Stack
• Responsible for creating reusable and scalable data pipelines.
• You are a Team-Player, collaborating with team members across multiple engineering teams to support the integration of proven prototypes into core intelligence products.
• You have strong communication skills to effectively convey complex data insights to non-technical stakeholders.
• You have an experience working collaboratively in cross-functional teams and manage multiple projects simultaneously

Skills

Databases

Big Data pipelines

Spark

Python

SQL

Azure Databricks

ETL pipelines

Azure Data Factory

Azure Synapse

Event Hub

Azure Data Lake Gen2

Power BI

Azure DevOps

Azure Kubernetes Service

Docker

Education

Bachelor’s degree in computer science, software engineering, information technology, or a related field

Tools

Databricks

Docker

Kubernetes

Power BI

Azure Synapse

Job description

Job Description:

You are detailed reviewing and analyzing structured, semi-structured and unstructured data sources for quality, completeness, and business value.

• You design, architect, implement and test rapid prototypes that demonstrate value of the data and present them to diverse audiences.
• You participate in early state design and feature definition activities.
• Responsible for implementing robust data pipeline using Microsoft, Databricks Stack
• Responsible for creating reusable and scalable data pipelines.
• You are a Team-Player, collaborating with team members across multiple engineering teams to support the integration of proven prototypes into core intelligence products.
• You have strong communication skills to effectively convey complex data insights to non-technical stakeholders.
• You have an experience working collaboratively in cross-functional teams and manage multiple projects simultaneously

Requirements

Skills:

• Advanced working knowledge and experience with relational and non-relational databases.
• Experience building and optimizing Big Data pipelines, architectures, and datasets.
• Strong analytic skills related to working with structured and unstructured datasets.
• Hands-on experience in Azure Databricks utilizing Spark to develop ETL pipelines.
• Strong proficiency in data analysis, manipulation, and statistical modeling using tools like Spark, Python, Scala, SQL, or similar languages.
• Strong experience in Azure Data Lake Storage Gen2, Azure Data Factory, Databricks, Event Hub, Azure Synapse.
• Familiarity with several of the following technologies: Event Hub, Docker, Azure Kubernetes Service, Azure DWH, API Azure, Azure Function, Power BI, Azure Cognitive Services.
• Azure DevOps experience to deploy the data pipelines through CI/CD.

Qualifications and Experience:

• Minimum 5-7 years of practical experience as Data Engineer.
• Bachelor’s degree in computer science, software engineering, information technology, or a related field.
• Azure cloud stack in-production experience.