Data Engineer

techcarrot; Telangana; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

techcarrot is seeking a data engineer to analyze structured, semi-structured and unstructured data sources for quality and business value; design, architect, implement and test rapid prototypes, and participate in early state design activities.

You will build robust data pipelines on the Azure Databricks stack, collaborate with multiple engineering teams, and drive integration of prototypes into core intelligence products using Azure DevOps.

Qualifications

• Advanced knowledge of relational and non-relational databases.
• Experience building and optimizing Azure data pipelines and datasets.
• Strong analytic skills with unstructured data.
• Experience with Azure Data Factory, Databricks, EventHub, Python, PySpark, Azure Synapse and SQL.
• Azure DevOps experience to deploy pipelines.

Responsibilities

• Review and analyze data sources for quality, completeness and value.
• Design, architect, implement and test rapid prototypes for diverse audiences.
• Participate in early state design and feature definition activities.
• Build robust, reusable data pipelines using Microsoft and Databricks stack.
• Collaborate with multiple engineering teams to integrate prototypes into core products.

Skills

Database experience

Data analysis

Team collaboration

Education

Bachelor’s degree

Tools

Azure Databricks

Azure Data Factory

Azure Synapse

Eventhub

Python

PySpark

SQL

Azure DevOps

Job description

Job Description:

• Detailed reviewing and analyzing structured, semi-structured and unstructured data sources for quality, completeness and business value.
• Design, architect, implement and test rapid prototypes that demonstrate value of the data and present them to diverse audiences.
• Participate in early state design and feature definition activities.
• Responsible for implementing robust data pipeline using Microsoft, Databricks Stack
• Responsible for creating reusable and scalable data pipelines
• You are a Team-Player, collaborating with team members across multiple engineering teams to support the integration of proven prototypes into core intelligence products.

Requirements

• Advanced working knowledge and experience with relational and non-relational databases
• Experience building and optimizing Azure Data data pipelines, architectures and data sets
• Strong analytic skills related to working with unstructured datasets
• Strong experience in Azure Data factory, Databricks, Eventhub, Python, PySpark , Azure Synapse and SQL, Azure DevOps
• Azure DevOps experience to deploy the ADF pipelines.
• Create end to end Azure databricks
• Minimum 4-6 years of experience
• Bachelor’s degree
• Knowledge/Experience with Azure cloud stack.
• Azure Data Platform & Databricks exp should be more than 4 years

Requirements