Sr Data Specialist - R01564133

Brillio; Bengaluru; On-site; INR 1,400,000 - 2,200,000; Full time

Job summary

Brillio Technologies in India is seeking a Senior Data Engineer with strong Databricks and Big Data expertise to design, build, and optimize scalable data pipelines and platforms.

You will work with Spark, PySpark, AWS ecosystem, and modern data architectures, including APIs and real-time processing, while collaborating with Analytics, Product and Engineering teams. Hybrid/remote work options are available.

Qualifications

• :minimum 5+ years of Data Engineering experience.

Responsibilities

• Design and build scalable data pipelines using Spark / PySpark on Databricks.
• Develop and optimize ETL/ELT workflows for large-scale data processing.
• Work with AWS EMR, S3, and Hadoop ecosystem for distributed data processing.
• Build and maintain data lake and data warehouse solutions.
• Develop and integrate APIs for data ingestion and consumption.
• Implement data processing workflows using Airflow / Autosys.
• Optimize data performance using partitioning, caching, and query tuning.
• Handle structured and semi-structured data from multiple sources.
• Ensure data quality, governance, and reliability of pipelines.
• Collaborate with cross-functional teams (Analytics, Product, Engineering).

Skills

Spark / PySpark

Databricks

SQL

Python

AWS

ETL fundamentals

APIs

Airflow

Autosys

Data modelling

Education

Bachelor's or Master's in CS/Engineering

Tools

AWS EMR

S3

Hadoop

Hive

Elasticsearch

Git

SVN

Job description

Lead Data Engineer

Primary Skills

• Athena, Step Functions, Spark - Pyspark, ETL Fundamentals, SQL (Basic + Advanced), Glue, Python, Lambda, Data Warehousing, EBS /EFS, AWS EC2, Lake Formation, Aurora, S3, Modern Data Platform Fundamentals, PLSQL, Data Modelling Fundamentals, Cloud front

Specialization

• AWS Data EngineerIng Basic: Senior Data Engineer

Job Requirements

Company: Brillio Technologies
Role: Senior Data Engineer
Experience: 5–10 Years
Location: India (Hybrid/Remote)
Employment Type: Full-time

Role Overview

We are looking for a Senior Data Engineer with strong Databricks and Big Data expertise to design, build, and optimize scalable data pipelines and platforms. The ideal candidate will have hands‑on experience with Spark, PySpark, AWS ecosystem, and modern data architectures, along with exposure to APIs and real‑time data processing.

Key Responsibilities

• Design and build scalable data pipelines using Spark / PySpark on Databricks
• Develop and optimize ETL/ELT workflows for large‑scale data processing
• Work with AWS EMR, S3, and Hadoop ecosystem for distributed data processing
• Build and maintain data lake and data warehouse solutions
• Develop and integrate APIs for data ingestion and consumption
• Implement data processing workflows using Airflow / Autosys
• Optimize data performance using partitioning, caching, and query tuning
• Handle structured and semi‑structured data from multiple sources
• Ensure data quality, governance, and reliability of pipelines
• Collaborate with cross‑functional teams (Analytics, Product, Engineering)

Technical Skills (Must-Have)

• Strong expertise in:
  • Apache Spark / PySpark
  • Databricks (mandatory)
  • SQL (PostgreSQL or similar)
• Hands‑on experience with:
  • AWS EMR, S3
  • Hadoop, Hive ecosystem
• Programming skills:
  • Python (must-have)
  • Scala (good exposure)
• Strong knowledge of:
  • UNIX / Shell scripting
  • ETL pipelines and data engineering fundamentals

Additional Skills

• Experience with:
  • Elasticsearch (data storage & retrieval)
  • Workflow orchestration tools (Airflow, Autosys)
• Exposure to:
  • API development & integration
  • Version control tools (Git / SVN)
• Basic knowledge of HTML (for web/data tasks)

Experience Required

• Minimum 5+ years of Data Engineering experience
• Strong hands‑on experience in Databricks + Big Data stack
• Proven experience working on large‑scale distributed systems

Education

• Bachelor’s or Master’s degree in Computer Science, Engineering, or related field

Why Join Brillio

• Work on cloud-native, large‑scale data platforms
• Exposure to cutting‑edge Databricks & AWS ecosystems
• Collaborative and high‑performance engineering culture