Senior Data Engineer

techcarrot; Hyderabad; On-site; INR 1,700,000 - 2,600,000; Full time

Job summary

techcarrot is seeking an experienced Data Engineer to design, build, and optimize scalable ingestion and analytical pipelines. You will work across multiple source systems, cleansing and transforming data for accurate reporting and dashboards.

The role requires strong Python/Scala/SQL skills, hands-on experience with Azure data services, and collaboration with cross-functional teams in an Agile environment. Hyderabad-based position with cloud-first projects.

Qualifications

• 5+ years of experience in Data Engineering and Analytics.
• Expertise in data ingestion, processing and analytical pipelines for big data, relational databases, and data warehouse solutions
• Hands-on experience with Agile software development
• Experience in designing and hands-on development in cloud-based analytics solutions.
• Expert level understanding on Azure Data Factory, Azure Synapse, Azure SQL, Azure Data Lake, and Azure App Service is required.
• Designing and building data pipelines using API ingestion and streaming ingestion methods.
• Knowledge of Dev-Ops processes (including CI/CD) and Infrastructure as code is essential.
• Thorough understanding of Azure and AWS Cloud Infrastructure offerings.
• Expertise in Azure Databricks, Azure Stream Analytics, Power BI is desirable
• Knowledge of SAP and BW/ BPC is desirable.
• Expert in Python, Scala, SQL is desirable
• Experience developing security models.

Responsibilities

• Design and develop scalable data ingestion and transformation pipelines integrating data from multiple source systems.
• Implement data cleansing, validation, and quality processes to ensure accurate, consistent, and reliable data.
• Build complex datasets and data models to support reporting, analytics, dashboards, and business decision-making.
• Analyze data and reporting requirements to identify trends, data gaps, and quality issues.
• Work closely with cross-functional teams using Agile methodology to deliver reliable and high-quality data solutions.

Skills

Python

Scala

SQL

Tools

Azure Data Factory

Azure Synapse

Azure SQL

Azure Data Lake

Azure App Service

Azure Databricks

Azure Stream Analytics

Power BI

SAP BW/BPC

Job description

Responsibilities

• Design and develop scalable data ingestion and transformation pipelines integrating data from multiple source systems.
• Implement data cleansing, validation, and quality processes to ensure accurate, consistent, and reliable data.
• Build complex datasets and data models to support reporting, analytics, dashboards, and business decision-making.
• Analyze data and reporting requirements to identify trends, data gaps, and quality issues.
• Work closely with cross-functional teams using Agile methodology to deliver reliable and high-quality data solutions.

Requirements

• At least 5 years of experience as an expert in Data Engineering and Analysis.
• Expertise in data ingestion, data processing and analytical pipelines for big data, relational databases, and data warehouse solutions
• Hands-on experience with Agile software development
• Experience in designing and hands-on development in cloud-based analytics solutions.
• Expert level understanding on Azure Data Factory, Azure Synapse, Azure SQL, Azure Data Lake, and Azure App Service is required.
• Designing and building of data pipelines using API ingestion and Streaming ingestion methods.
• Knowledge of Dev-Ops processes (including CI/CD) and Infrastructure as code is essential.
• Thorough understanding of Azure and AWS Cloud Infrastructure offerings.
• Expertise in Azure Databricks, Azure Stream Analytics, Power BI is desirable
• Knowledge of SAP and BW/ BPC is desirable.
• Expert in Python, Scala, SQL is desirable
• Experience developing security models.