Senior Data Engineer

Expian Technologies; Bengaluru; Hybrid; INR 1,000,000 - 1,500,000; Full time

Job summary

Expian Technologies is looking for a skilled Data Engineer in Bengaluru, India. The role requires designing and implementing scalable data solutions using GCP and Databricks.

The ideal candidate will have over 5 years of experience, with strong expertise in Python, SQL, and modern data architectures.

This position offers an opportunity to work on exciting projects with cross-functional teams, focusing on efficient ETL processes and data-driven product development.

Qualifications

• 5+ years of experience as a Data Engineer, designing and building scalable data pipelines.
• Strong expertise in Python and SQL, including data transformation and performance optimization.
• Experience with Google Cloud Platform services like BigQuery, Cloud Storage is mandatory.

Responsibilities

• Design, implement scalable data solutions using GCP and Databricks.
• Maintain modern data pipelines and manage ETL processes.
• Establish best practices for data testing and validation.

Skills

Python

SQL

Databricks

Google Cloud Platform (GCP)

ETL/ELT pipelines

Apache Airflow

Docker

Kubernetes

Tools

BigQuery

PostgreSQL

Snowflake

Redshift

Job description

Roles and Responsibilities

• Plan, design, and implement scalable data solutions using GCP and Databricks technologies.
• Design, develop, and maintain modern data pipelines and real-time data streaming solutions.
• Manage and optimise ETL processes to ensure efficient data processing, transformation, and integration.
• Develop and maintain data marts to support defined business and data-driven use cases.
• Pre-process, clean, and transform structured and unstructured data, while creating and optimising queries as required.
• Establish and implement best practices, frameworks, and standards for data testing, validation, and quality assurance.
• Contribute to the development of data-driven products, including user journey analysis and editorial intelligence tools.
• Collaborate closely with cross-functional technology teams to align data initiatives with business and technical objectives.
• Perform root cause analysis on internal and external data issues to solve business problems and identify improvement opportunities.
• Build and maintain processes supporting data transformation, metadata management, dependency handling, and workload optimisation.
• Analyse, manipulate, and extract actionable insights from large and complex datasets across multiple disconnected sources.

Key Requirements

• 5+ years of hands‑on experience as a Data Engineer, designing and building scalable data pipelines and cloud‑based data processing solutions.
• Strong hands‑on expertise inPython and SQL, including data transformation, performance optimisation, data modelling, workflow automation and large‑scale data processing.
• Experience working withGoogle Cloud Platform (GCP) services such asBigQuery, Cloud Storage and Cloud Functions, or equivalent cloud‑native data platforms, is mandatory.
• Hands‑on experience withDatabricks, including notebooks, clusters, Spark/PySpark development, data transformation, and pipeline orchestration.
• Strong understanding of distributed data processing frameworks, particularlyPySpark/Spark DataFrames, for large‑scale data transformation and analytics workloads.
• Experience working with relational databases such asPostgreSQL, SQL Server, Oracle or similar platforms, including complex SQL development, query optimisation and data modelling.
• Proven experience developing and maintainingETL/ELT pipelines, including ingestion from APIs, databases, cloud storage platforms and third‑party data sources.
• Understanding of modern data architecture patterns such asMedallion Architecture (Bronze/Silver/Gold), Data Lake, Data Warehouse, and Lakehouse architectures.
• Experience working with version control systems and CI/CD pipelines using tools such asGitHub Actions, GitLab CI/CD, Azure DevOps or equivalent platforms.
• Experience working with modern data processing and analytics platforms such asDatabricks, Snowflake, Redshift, Synapse Analytics or similar technologies is desirable.
• Experience designing and implementing reusable Python libraries, utility packages, frameworks, and automation solutions for data engineering workflows.
• Familiarity with cloud‑based orchestration and workflow management tools such asApache Airflow, Dataform,Azure components - Azure Data Lake, Azure SQL DW, Azure Synapse, etc. or equivalent platforms is good to have.
• Experience integrating and processing data from diverse sources, includingAPIs, cloud storage, structured databases and file formats such as JSON, CSV and Parquet.
• Exposure to containerization and cloud‑native deployment technologies such as Docker and Kubernetes would be an advantage.
• Any understanding of Web Analytics data, such as Adobe Analytics and Google Analytics, alongside any Ad tech like Google Ad Manager (working with programmatic and affiliate partners), is a plus, but not essential.
• Strong problem‑solving skills with the ability to independently analyse business requirements, propose scalable data solutions, and collaborate effectively with analysts and stakeholders.