Data Specialist - R01569712

Brillio; Bengaluru; On-site; INR 2,800,000 - 4,200,000; Full time

Job summary

Brillio in Bengaluru is seeking a Lead Data Engineer with deep expertise in Databricks and PySpark to architect and optimize large-scale data pipelines. You will design end-to-end ETL, build scalable data models, and collaborate with architects and stakeholders to deliver analytics-ready data solutions.

The role requires 5-6 years of experience in data engineering, strong SQL, and cloud familiarity, with a focus on data warehousing and Databricks platform best practices.

Qualifications

• 5-6 years of data engineering experience with Databricks and PySpark.
• Strong SQL skills for analytics and data modeling.
• Experience with data warehousing architectures.
• Ability to translate requirements into scalable data solutions.
• Experience with cloud platforms (Azure/AWS) is a plus.

Responsibilities

• Design and implement robust ETL pipelines using Databricks and PySpark.
• Develop and optimize SQL queries, procedures, and data models.
• Collaborate with data architects and business stakeholders.
• Integrate data from diverse sources into data warehouses.
• Troubleshoot data ingestion, transformation, and storage issues in Databricks.
• Enforce best practices in data warehousing and Databricks fundamentals.
• Monitor and optimize performance of data processing jobs.
• Document processes and share best practices with the team.

Skills

Databricks

PySpark

SQL

Education

Bachelor's degree in Computer Science

Job description

Primary Skills

Databricks, Pyspark, sql

Specialization

• Databricks Engineering: Lead Data Engineer

Job requirements

Experience Range: With at least 5-6 years of experience in data engineering, including substantial hands-on expertise in Databricks and PySpark

• 1. Design and implement robust ETL pipelines using Databricks and PySpark to efficiently process large-scale datasets
• 2. Develop, optimize, and maintain complex SQL queries, stored procedures, and scalable data models to support analytics and reporting
• 3. Collaborate with data architects and business stakeholders to translate requirements into high-performance data solutions
• 4. Integrate data from diverse sources into data warehouses and modern data platforms, ensuring data quality and consistency
• 5. Troubleshoot and resolve issues related to data ingestion, transformation, and storage within Databricks environments
• 6. Enforce best practices in data warehousing, data modeling, and Databricks platform fundamentals
• 7. Monitor and tune performance of data processing jobs for optimal throughput and reliability
• 8. Document data engineering processes and provide technical guidance to team members on Databricks and PySpark usage

• 1. Advanced proficiency in Databricks platform
• 2. Expertise in PySpark for distributed data processing and transformation
• 3. Strong SQL skills for analytics and data manipulation
• 4. Experience with data warehousing architectures
• 5. Hands-on experience designing and maintaining stored procedures
• 6. Proficiency in Apache Spark for large-scale data processing

• 1. Experience with Azure Databricks or AWS Glue
• 2. Familiarity with CI/CD pipelines for Databricks workflows
• 3. Knowledge of Databricks performance optimization techniques
• 4. Exposure to data governance and security best practices in cloud environments
• 5. Experience with real-time data streaming frameworks such as Apache Kafka

• 1. Bachelor's degree in Computer Science, Information Technology, or a closely related discipline
• 2. Certification in Databricks or Apache Spark
• 3. Certification in SQL or data warehousing technologies