Data Engineering Specialist

BestPeers; Indore District; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

BestPeers is seeking a Data Engineering Specialist to build and support reliable data solutions for BI, analytics, and AI use cases in Indore. You will design scalable pipelines using Python, SQL, and cloud platforms.

You will collaborate with BI and analytics teams, implement data quality and governance, and optimize workflows with Airflow, dbt, and modern data tools. Strong problem-solving and communication skills are essential.

Qualifications

• Strong Python and SQL skills with production-grade data solutions.
• Experience building ETL/ELT pipelines in production environments.
• Hands-on cloud data engineering on AWS, GCP, or Azure.
• Experience with BI platforms and analytics reporting.
• Knowledge of data warehousing and dimensional modeling.
• Experience with Airflow, Data Factory, or Step Functions.
• Knowledge of data quality, governance, security, and production support.
• Strong problem-solving and collaboration skills.

Responsibilities

• Design, build, and maintain scalable data pipelines across cloud platforms.
• Develop data solutions using Python, SQL, and distributed processing technologies.
• Integrate data from APIs, databases, files, and event-based sources.
• Build analytics-ready datasets and data models for BI workloads.
• Develop and maintain workflows using Airflow or similar orchestration platforms.
• Work closely with BI and analytics teams to translate requirements into datasets and reporting solutions.
• Support BI platforms such as Power BI, Tableau, or QuickSight, including data models and validation.
• Develop efficient SQL transformations and optimize queries, pipelines, and data models.
• Implement data quality, validation, monitoring, alerting, and error-handling processes.
• Investigate production issues and data discrepancies, perform root-cause analysis, and drive resolution.
• Follow engineering, security, governance, and privacy best practices with documentation.

Skills

Python

SQL

ETL/ELT pipelines

Cloud data engineering

Airflow

dbt

Git

Docker

Tools

Airflow

dbt

Git

Docker

Power BI

Tableau

Job description

We are looking for a Data Engineering Specialist to build and support reliable data solutions for

BI, analytics, and AI use cases. The ideal candidate has strong Python and SQL skills, hands-on experience with data pipelines and cloud platforms, and experience working with BI and analytics platforms.

Key Responsibilities

• Design, build, and maintain scalable batch and streaming data pipelines across cloud-based data platforms.
• Develop data solutions using Python, SQL, and distributed processing technologies such as PySpark.
• Integrate data from APIs, databases, files, and event-based sources.
• Build reliable, analytics-ready datasets and data models for BI and analytical workloads.
• Develop and maintain workflows using Airflow or similar orchestration platforms.
• Work closely with BI and analytics teams to translate business requirements into trusted datasets and reporting solutions.
• Support BI platforms such as Power BI, Tableau, or QuickSight, including data models, reporting datasets, metrics, and data validation.
• Develop efficient SQL transformations and optimize queries, pipelines, and data models for performance, scalability, and cost.
• Implement data quality, validation, monitoring, alerting, and error-handling processes.
• Investigate production issues and data discrepancies, perform root-cause analysis, and drive resolution.
• Follow engineering, security, governance, and privacy best practices while maintaining appropriate technical documentation and operational runbooks.

Required Skills

• Strong Python and SQL programming skills with experience developing production-grade data solutions.
• Proven experience building and supporting ETL/ELT pipelines in production environments.
• Hands-on experience with cloud data engineering on AWS, GCP, Azure, or similar platforms.
• Experience working with cloud storage, data processing technologies, and cloud data warehouses.
• Good understanding of data warehousing and dimensional modeling, including facts, dimensions, and slowly changing dimensions.
• Experience with a workflow orchestration platform such as Airflow, Azure Data Factory, AWS Step Functions, or similar.
• Experience working with BI and analytics platforms, with an understanding of how data engineering solutions support reporting and business analytics.
• Experience with data transformation and testing frameworks such as dbt or similar tools.
• Working knowledge of Git, CI/CD, and Docker.
• Understanding of data quality, monitoring, governance, security, and production support.
• Strong problem-solving, troubleshooting, communication, and collaboration skills.

Good to Have

• Hands-on experience with Apache Airflow and Power BI.
• Experience with Kafka, Kinesis, CDC, Debezium, or Spark Structured Streaming.
• Experience with Databricks or other modern data platforms.
• Knowledge of Terraform and cloud security.
• Experience developing lightweight APIs using FastAPI, Flask, Django, or similar frameworks.
• Experience supporting data platforms for AI/ML or LLM applications.
• Experience working with regulated or sensitive data.
• Relevant cloud or data engineering certifications.
• Experience using AI/LLM tools to improve engineering productivity.