Senior Data Engineer

CloudBoson; Dadri; On-site; INR 1,800,000 - 3,000,000; Full time

Job summary

CloudBoson Technologies Private Limited is hiring a data engineer to design, build, and optimize ETL/ELT pipelines using PySpark on Azure Databricks, and to develop real-time ingestion pipelines with Spark Structured Streaming and Azure Event Hubs.

You will work with BigQuery for analytics and Looker Studio for reporting, collaborating with product teams and mentoring engineers while enforcing CI/CD practices for data pipelines.

Qualifications

• 2+ years in data engineering.
• Strong SQL with joins, window functions and optimization.
• Proficient in Python and PySpark (Azure Databricks).
• Experience with real-time processing and multi-cloud (GCP + Azure).
• Looker Studio for BI/reporting and Git version control.
• REST API integration experience.

Responsibilities

• Design, build, and optimize ETL/ELT pipelines using PySpark on Azure Databricks.
• Develop real-time and near-real-time ingestion pipelines using Spark Structured Streaming and Azure Event Hubs.
• Develop and tune SQL models in BigQuery for analytics and reporting.
• Build ingestion pipelines from APIs and third-party sources into Azure Data Lake/BigQuery.
• Implement incremental loads, CDC, and MERGE/upsert patterns for large datasets.
• Monitor data pipelines and optimize cost & performance of queries.
• Co-ordinate with stakeholders to provide data and insights.

Skills

SQL

Python

PySpark

Azure Databricks

Spark Structured Streaming

BigQuery

Looker Studio

Git

REST API integration

Data modeling

Tools

Azure Event Hubs

Azure Data Lake

ADLS Gen2

Azure Storage Tables

Azure SQL

BigQuery

Airflow

Job description

Role & responsibilities

• Design, build, and optimize ETL/ELT pipelines using PySpark on Azure Databricks.
• Develop real-time and near-real-time ingestion pipelines using Spark Structured Streaming and Azure Event Hubs
• Develop and tune complex SQL models in BigQuery for analytics and reporting
• Build ingestion pipelines from APIs and third-party sources into Azure Data Lake / BigQuery
• Implement incremental loads, CDC, and MERGE/upsert patterns for large datasets.
• Experience on data lake/warehouse solutions across GCP and Azure
• Build dashboards and reporting layers in Looker Studio
• Lead code reviews, mentor engineers, and drive Git-based CI/CD practices for data pipelines
• Partner with product/business teams to translate requirements into scalable data models
• Optimize cost & performance of already running queries
• Carefully plan and implement data load requirement.
• Regular monitoring of the data pipelines and systems and maintains the uptime of the overall data availability.
• Co-ordinate with stake holders and provide relevant data and business insights.

Required Skills:

• 2+ years in data engineering
• Strong SQL (joins, window functions, query optimization)
• Well versed in Python and PySpark (Azure Databricks)
• Spark Structured Streaming for real-time data processing
• Azure Event Hubs for event ingestion and streaming architecture
• BigQuery , AWS DynamoDB, Azure Storage Tables, Azure SQL, Azure Event Hub.
• Azure Data Lake (ADLS Gen2)
• Multi-cloud experience: GCP + Azure
• Looker Studio for BI/reporting
• Git version control
• REST API integration (auth, pagination, error handling)

Good to have:

• Orchestration tools (Airflow, Databricks Workflows)
• Data quality/governance frameworks
• Cloud cost optimization experience