Data Engineer

Axi; Bengaluru; On-site; INR 1,500,000 - 3,000,000; Full time

Job summary

Axi Bengaluru data engineering team builds scalable data platforms on Azure and Databricks, partnering with global Risk, Finance, and Analytics teams. You will develop end-to-end data pipelines that ingest, transform, and serve data with strict quality and SLA requirements.

Responsibilities include building ETL/ELT pipelines on Azure Databricks using PySpark or Scala, ingesting data from SQL Server, Cosmos DB, event streams, APIs, and external sources, and implementing data quality checks and

Qualifications

• 2-6 years of data engineering or analytics engineering experience.
• Strong problem-solving abilities and independent work with guidance.
• Good communication to translate business needs into technical data solutions.
• Experience in Agile environments and collaborating across time zones.
• Hands-on experience with Azure Databricks and Apache Spark (PySpark/Scala).
• Strong SQL skills with complex queries, window functions, and optimization.
• Experience with SQL Server and data warehousing concepts.
• Familiarity with Azure data services and metadata management.
• Understanding of ETL/ELT concepts and dimensional modeling.
• Experience with batch/stream processing and event-driven architectures.
• Knowledge of data quality frameworks and monitoring.

Responsibilities

• Build and maintain ETL/ELT pipelines on Azure Databricks using PySpark or Scala.
• Ingest data from SQL Server, Cosmos DB, event streams, APIs into cloud data lakes and lakehouses.
• Design data quality checks, validation rules, and monitoring for reliability.
• Optimise Spark jobs for performance and cost, including clustering and caching.
• Document pipelines, lineage, and schemas; maintain metadata catalogs.
• Collaborate with data analysts, scientists, and domain engineers to refine data products.
• Contribute to infrastructure and tooling for data workflows (scheduling, orchestration, monitoring, alerts).

Skills

Problem solving

Communication

Independent work

Agile collaboration

Data modeling

Dimensional modeling

ETL/ELT concepts

Data quality mindset

Monitoring & testing

Tools

Azure Databricks

Apache Spark

PySpark

Scala

SQL Server

Cosmos DB

Azure Data Lake Storage

Azure Synapse

Azure Data Factory

Data Factory

Databricks lakehouses

Job description

The Data Engineering team builds and operates robust, scalable data platforms on Azure and Databricks, closely partnering with domain PODs, Risk, Finance, and Analytics teams across the globe. You'll work on end-to-end data pipelines that ingest, transform, and serve data to multiple downstream consumers with strict quality and SLA requirements.

Responsibilities:

• Build and maintain ETL/ELT pipelines on Azure Databricks using PySpark or Scala to ingest, transform, and deliver data products at scale.
• Ingest data from SQL Server, Cosmos DB, event streams, APIs, and external sources into cloud data lakes and Databricks lakehouses.
• Design and implement data quality checks, validation rules, and monitoring to ensure reliability and early detection of anomalies.
• Optimise Spark jobs for performance and cost, including cluster sizing, partitioning strategies, and caching mechanisms.
• Document data pipelines, lineage, and schemas; maintain data dictionaries and metadata catalogues for consumer discovery and understanding.
• Collaborate with data analysts, scientists, and domain engineers to understand requirements and refine data products.
• Contribute to infrastructure and tooling for data workflows (scheduling, orchestration, monitoring, alerting).

Requirements:

• 2-6 years of experience in data engineering or analytics engineering roles.
• Strong problem-solving and ability to work independently with guidance from senior engineers.
• Good communication skills to translate business requirements into technical data solutions.
• Ability to work in Agile environments and collaborate with global teams across time zones
• Hands-on experience with Azure Databricks and Apache Spark (PySpark or Scala) for large-scale data processing.
• Strong SQL skills, including complex queries, window functions, CTEs, and query optimisation for both transactional and analytical workloads.
• Experience with SQL Server: data warehousing concepts, schema design, indexing, and optimisation for analytics.
• Familiarity with Azure data services: Data Lake Storage, Synapse Analytics, Data Factory, or Blob Storage.
• Understanding of ETL/ELT concepts, data modelling, dimensional modelling (fact/dimension tables), and slowly changing dimensions.
• Experience with batch and/or stream processing concepts; familiarity with event-driven architectures.
• Knowledge of data quality frameworks, testing strategies, and monitoring/alerting for data pipelines.
• Basic understanding of relational and NoSQL databases (Cosmos DB exposure is beneficial).