Data Engineer

Techcarrot Fz-llc; Hyderabad; On-site; INR 1,500,000 - 2,100,000; Full time

Job summary

Techcarrot Fz-llc in Hyderabad, India is seeking a Data Engineer to design, build and optimize scalable data pipelines on Azure. You will work across Azure Data Factory, Databricks, Delta Lake and storage services to deliver end-to-end solutions.

The role requires strong PySpark, SQL and Python skills, experience with CI/CD, and the ability to operate across data sources and pipelines, ensuring quality and reliability in production environments.

Qualifications

• 46 years of overall experience in Data Engineering / Data Platform development.
• Preferably 4+ years of hands-on experience with Microsoft Azure Data Platform technologies.
• Strong hands-on experience with Azure Data Factory, Databricks, SQL and Python/PySpark.
• Bachelors degree in Computer Science, Information Technology, Engineering or a related discipline.
• Experience delivering at least one or more end-to-end Azure data engineering projects.
• Good understanding of data warehousing, data lake and modern data platform concepts.
• Strong analytical and problem-solving skills.
• Ability to work independently and manage multiple technical activities.
• Good written and verbal communication skills.

Responsibilities

• Independently understand business and technical requirements and translate them into scalable data engineering solutions.
• Take end-to-end ownership of assigned projects, modules and data pipelines with minimal supervision.
• Design, develop, test and deploy robust end-to-end data pipelines on the Azure Data Platform.
• Build scalable and reusable data ingestion and transformation frameworks using Azure Data Factory and Azure Databricks.
• Develop data transformation and processing logic using Python, PySpark and SQL.
• Work with structured, semi-structured and unstructured data from databases, APIs, files and other enterprise data sources.
• Design and implement batch and, where required, near-real-time data processing solutions.
• Develop optimized Databricks workloads using Apache Spark and Delta Lake.
• Perform data analysis, profiling and validation to identify data quality, completeness and consistency issues.
• Implement appropriate data quality checks, error handling, logging and monitoring within data pipelines.
• Troubleshoot data pipeline failures, performance issues and data discrepancies independently.
• Optimize SQL queries, Spark jobs and data pipelines for performance and scalability.
• Create reusable components and frameworks to improve development efficiency across projects.
• Participate in solution design and technical discussions and provide recommendations on implementation approaches.
• Implement CI/CD processes for data engineering components using Azure DevOps.
• Deploy and manage ADF, Databricks and related Azure data components across development, test and production environments.
• Prepare appropriate technical design, data mapping and operational documentation.
• Collaborate with solution architects, analysts, BI developers, source-system teams and other engineering teams.
• Support production deployments and troubleshoot post-production issues when required.

Skills

Azure Data Factory

Databricks

SQL

Python

PySpark

CI/CD

Azure DevOps

Delta Lake

Spark

REST APIs

Power BI

Unity Catalog

ETL/ELT

Education

Bachelor's degree in Computer Science/IT/Engineering

Tools

Azure Databricks

Azure Data Factory

ADLS Gen2

Azure Synapse

Azure Monitor

Job description

Responsibilities

• Independently understand business and technical requirements and translate them into scalable data engineering solutions.
• Take end-to-end ownership of assigned projects, modules and data pipelines with minimal supervision.
• Design, develop, test and deploy robust end-to-end data pipelines on the Azure Data Platform.
• Build scalable and reusable data ingestion and transformation frameworks using Azure Data Factory and Azure Databricks.
• Develop data transformation and processing logic using Python, PySpark and SQL.
• Work with structured, semi-structured and unstructured data from databases, APIs, files and other enterprise data sources.
• Design and implement batch and, where required, near-real-time data processing solutions.
• Develop optimized Databricks workloads using Apache Spark and Delta Lake.
• Perform data analysis, profiling and validation to identify data quality, completeness and consistency issues.
• Implement appropriate data quality checks, error handling, logging and monitoring within data pipelines.
• Troubleshoot data pipeline failures, performance issues and data discrepancies independently.
• Optimize SQL queries, Spark jobs and data pipelines for performance and scalability.
• Create reusable components and frameworks to improve development efficiency across projects.
• Participate in solution design and technical discussions and provide recommendations on implementation approaches.
• Implement CI/CD processes for data engineering components using Azure DevOps.
• Deploy and manage ADF, Databricks and related Azure data components across development, test and production environments.
• Prepare appropriate technical design, data mapping and operational documentation.
• Collaborate with solution architects, analysts, BI developers, source-system teams and other engineering teams.
• Support production deployments and troubleshoot post-production issues when required.

Requirements

• Strong hands-on experience with:
  • Azure Data Platform
    • Azure Data Factory (ADF) pipelines, datasets, linked services, triggers, parameterization and reusable frameworks
    • Azure Databricks
    • Azure Data Lake Storage Gen2 (ADLS Gen2)
    • Azure Synapse Analytics / Azure SQL
    • Azure Key Vault
    • Azure Monitor / Log Analytics or equivalent monitoring capabilities
    • Azure Event Hubs exposure/experience is preferred
  • Databricks & Data Processing
    • Strong hands-on experience with Databricks and Apache Spark
    • Strong PySpark development skills
    • Experience with Delta Lake / Delta tables
    • Understanding of Medallion or similar layered data architectures
    • Experience implementing incremental and full-load processing patterns
    • Ability to troubleshoot and optimize Spark workloads
    • Experience with Databricks Workflows/Jobs
    • Knowledge of Unity Catalog is preferred
  • SQL & Python
    • Strong SQL development skills
    • Ability to write and optimize complex SQL queries
    • Experience with joins, CTEs, window functions and analytical queries
    • Strong working knowledge of Python
    • Ability to develop reusable Python/PySpark modules and utilities
  • DevOps / CI-CD
    • Hands-on experience with Azure DevOps
    • Git-based source control
    • Branching and code-management practices
    • CI/CD implementation for Azure Data Factory and Databricks
    • Experience deploying solutions across multiple environments

  Qualifications & Experience

  • 46 years of overall experience in Data Engineering / Data Platform development.
  • Preferably 4+ years of hands-on experience with Microsoft Azure Data Platform technologies.
  • Strong hands-on experience with Azure Data Factory, Databricks, SQL and Python/PySpark.
  • Bachelors degree in Computer Science, Information Technology, Engineering or a related discipline.
  • Experience delivering at least one or more end-to-end Azure data engineering projects.
  • Good understanding of data warehousing, data lake and modern data platform concepts.
  • Strong analytical and problem-solving skills.
  • Ability to work independently and manage multiple technical activities.
  • Good written and verbal communication skills.

  Preferred Skills

  • Microsoft Azure / Databricks certifications.
  • Experience with Unity Catalog and Databricks governance.
  • Experience with REST API-based data ingestion.
  • Knowledge of dimensional modelling and data warehouse concepts.
  • Experience working with Power BI or downstream analytics platforms.
  • Exposure to streaming/event-driven data processing.
  • Experience working in enterprise-scale Azure environments.

  Job Details

  • Job Type: Full time
  • Job Opening Name: Data Engineer
  • State: Telangana
  • Country: India
  • Location: Hyderabad, Noida/ Chennai