Job Description – Senior Data Engineer (Databricks & Apache Spark)Job Title: Senior Data EngineerExperience Required: 5+ YearsIndustry/Domain: Insurance / BFSIWe are seeking a highly skilled and experienced Senior Data Engineer to design, develop, and implement end-to-end data pipelines and enterprise data warehousing solutions for a leading insurance enterprise client. The ideal candidate should have strong expertise in Databricks, Apache Spark, SQL, data integration, and database performance optimization, along with a solid understanding of data warehousing concepts and business intelligence tools.
Key Responsibilities
- Design, develop, and maintain scalable end-to-end data pipelines for enterprise data integration and analytics.
- Build and optimize ETL/ELT workflows using Databricks and Apache Spark.
- Design and implement robust data warehousing solutions to support enterprise reporting and business intelligence requirements.
- Integrate data from multiple structured and semi-structured sources while ensuring data quality, consistency, and reliability.
- Develop complex SQL queries, stored procedures, and data transformation logic.
- Perform SQL query optimization, database performance tuning, and troubleshooting.
- Collaborate with data architects, business analysts, and stakeholders to understand data requirements and deliver efficient solutions.
- Support data modeling, data validation, and data governance best practices.
- Enable seamless data consumption through BI and reporting tools such as Power BI and Tableau.
- Monitor, troubleshoot, and enhance existing data pipelines to improve performance and scalability.
- Work with cross-functional teams to deliver data engineering solutions aligned with insurance business objectives.
Required Skills & Qualifications
- 5+ years of hands-on experience in data engineering, data integration, ETL/ELT development, or data warehousing.
- Strong practical experience with Databricks and Apache Spark, including distributed data processing.
- Advanced proficiency in SQL, including complex queries, query optimization, and database performance tuning.
- Strong understanding of data warehousing architecture, dimensional modeling, and ETL/ELT frameworks.
- Experience designing and implementing scalable, high-performance data pipelines.
- Familiarity with Power BI and Tableau for reporting, visualization, and analytics.
- Good understanding of relational databases, data transformation, and data integration methodologies.
- Strong analytical, problem-solving, and troubleshooting skills.
- Excellent communication skills and the ability to collaborate with technical and business stakeholders.
Preferred / Good-to-Have Skills
- Experience working with insurance or BFSI enterprise clients.
- Knowledge of insurance-related data, including policy, claims, underwriting, and customer information.
- Experience with Delta Lake, PySpark, and Python.
- Familiarity with cloud platforms such as Azure, AWS, or GCP.
- Understanding of data governance, data security, and regulatory compliance.
- Experience with workflow orchestration, CI/CD, and automated data pipeline deployment.
Interested in this role?