Data Specialist – R01569712

Brillio; Bengaluru; On-site; INR 3,500,000 - 5,500,000; Full time

Job summary

Brillio Bengaluru seeks an experienced Data Engineer to design and optimize end-to-end data pipelines using Databricks and PySpark. You will develop scalable data models, write complex SQL, and collaborate with architects and stakeholders to deliver high-performance analytics solutions.

The role emphasizes data quality, governance, and efficient processing of large-scale datasets in cloud environments, with exposure to Spark, Databricks, and modern data platforms.

Qualifications

• 5–6 years of data engineering experience with Databricks and PySpark
• Hands-on expertise in Databricks platform and Spark ecosystem
• Strong SQL, data modeling, and data warehousing knowledge

Responsibilities

• Design and implement robust ETL pipelines using Databricks and PySpark to efficiently process large-scale datasets
• Develop, optimize, and maintain complex SQL queries, stored procedures, and scalable data models to support analytics and reporting
• Collaborate with data architects and business stakeholders to translate requirements into high-performance data solutions
• Integrate data from diverse sources into data warehouses and modern data platforms, ensuring data quality and consistency
• Troubleshoot and resolve issues related to data ingestion, transformation, and storage within Databricks environments
• Enforce best practices in data warehousing, data modeling, and Databricks platform fundamentals
• Monitor and tune performance of data processing jobs for optimal throughput and reliability
• Document data engineering processes and provide technical guidance to team members on Databricks and PySpark usage

Skills

Databricks

PySpark

SQL

Data warehousing

Stored procedures

Apache Spark

Azure Databricks

AWS Glue

CI/CD

Databricks performance

Data governance

Kafka

Education

Bachelor's degree in Computer Science/IT

Databricks certification

SQL or data warehousing certification

Job description

Job requirements

Experience Range: With at least 5-6 years of experience in data engineering, including substantial hands-on expertise in Databricks and PySpark

• 1. Design and implement robust ETL pipelines using Databricks and PySpark to efficiently process large-scale datasets
• 2. Develop, optimize, and maintain complex SQL queries, stored procedures, and scalable data models to support analytics and reporting
• 3. Collaborate with data architects and business stakeholders to translate requirements into high-performance data solutions
• 4. Integrate data from diverse sources into data warehouses and modern data platforms, ensuring data quality and consistency
• 5. Troubleshoot and resolve issues related to data ingestion, transformation, and storage within Databricks environments
• 6. Enforce best practices in data warehousing, data modeling, and Databricks platform fundamentals
• 7. Monitor and tune performance of data processing jobs for optimal throughput and reliability
• 8. Document data engineering processes and provide technical guidance to team members on Databricks and PySpark usage

• 1. Advanced proficiency in Databricks platform
• 2. Expertise in PySpark for distributed data processing and transformation
• 3. Strong SQL skills for analytics and data manipulation
• 4. Experience with data warehousing architectures
• 5. Hands-on experience designing and maintaining stored procedures
• 6. Proficiency in Apache Spark for large-scale data processing

• 1. Experience with Azure Databricks or AWS Glue
• 2. Familiarity with CI/CD pipelines for Databricks workflows
• 3. Knowledge of Databricks performance optimization techniques
• 4. Exposure to data governance and security best practices in cloud environments
• 5. Experience with real-time data streaming frameworks such as Apache Kafka

• 1. Bachelor's degree in Computer Science, Information Technology, or a closely related discipline
• 2. Certification in Databricks or Apache Spark
• 3. Certification in SQL or data warehousing technologies