Databricks Data Engineer

Customized Energy Solutions; Pune District; Ibrido; INR 1.000.000 - 1.600.000; Tempo pieno

Vantaggi offerti da questo lavoro

Hybrid work arrangement

Office in Pune

Descrizione del lavoro

CES in Pune, India is hiring a Databricks Data Engineer to lead data modernization efforts. The role focuses on building scalable data pipelines with Databricks, Spark, and Delta Lake, migrating a large SQL Server data warehouse to the Databricks Lakehouse.

The ideal candidate will have 4–7 years in data engineering, strong Spark/SQL skills, and experience migrating on‑prem warehouses to cloud platforms. Hybrid work with three days/week in the Pune office is preferred.

Competenze

• Bachelor’s degree in Computer Science, Engineering, or a related field.
• 4–7 years of overall data engineering experience.
• Hands-on Databricks on AWS preferred; Azure Databricks acceptable.
• Strong Spark (PySpark/Scala) and SQL skills.
• Experience migrating on‑prem SQL Server warehouses to cloud platforms.
• Experience converting SSIS ETL to Spark-based solutions.
• Experience in enterprise modernization initiatives, not just lift-and-shift.

Mansioni

• Design, build, and optimize scalable data pipelines using Databricks, Spark, Delta Lake, and Unity Catalog.
• Migrate approximately 20TB on‑prem SQL Server data warehouse to Databricks.
• Convert SSIS ETL pipelines to Databricks notebooks and workflows.
• Re‑engineer SSAS analytics to Databricks SQL and Delta tables.
• Implement data quality, validation, and migration controls.
• Optimize Spark performance and cost efficiency on large datasets.
• Apply governance, security, lineage, and access controls with Unity Catalog.
• Collaborate with analytics/BI/AI teams for reporting and use cases.
• Contribute to reusable frameworks and platform best practices.
• Document platform architecture and engineering standards.

Conoscenze

Spark (PySpark/Scala)

SQL

Databricks AWS/Azure

Data pipelines

Delta Lake

Unity Catalog

Data warehousing

SSIS migration

Formazione

Bachelor's degree in Computer Science/Engineering

Strumenti

Databricks

Delta Lake

Power BI

Descrizione del lavoro

We are seeking a Databricks Data Engineer to join our growing data engineering team in Pune, India. CES is open to considering qualified candidates for either a direct-hire, full-time opportunity or a contract-to-hire arrangement, depending on the candidate's experience, availability, and preferred employment structure.

This role will play a key part in a large-scale modernization initiative to migrate a complex, enterprise-grade Microsoft SQL Server data warehouse ecosystem to the Databricks Lakehouse Platform.

The ideal candidate has strong hands-on experience across Databricks data engineering capabilities, with a core focus on building scalable, reliable data pipelines and analytics workloads. Exposure to Databricks AI/ML capabilities is a plus.

Key Responsibilities

• Design, build, and optimize scalable data pipelines using Databricks, Apache Spark, Delta Lake, and Unity Catalog.
• Participate in the migration of an approximately 20TB on-premises Microsoft SQL Server data warehouse to Databricks.
• Convert and modernize hundreds of SQL Server tables, thousands of SSIS jobs, and downstream SSRS and SSAS workloads.
• Re-engineer legacy SSIS ETL processes into Databricks notebooks, workflows, and orchestration frameworks.
• Support the migration and redesign of SSAS cube-based analytics into Databricks SQL, Delta tables, and modern semantic models.
• Implement data quality, validation, reconciliation, audit, and migration controls throughout the modernization process.
• Optimize Spark performance, partitioning strategies, query performance, and cost efficiency across large-scale datasets.
• Apply data governance, security, lineage, and access-control standards using Unity Catalog.
• Collaborate with analytics, BI, and AI/ML teams to enable downstream reporting, analytics, and advanced data use cases.
• Contribute to reusable frameworks, technical documentation, engineering standards, and platform best practices.

Qualifications

• Bachelor's degree in Computer Science, Engineering, or a related field.
• 4-7 years of overall data engineering experience.
• Hands-on experience with Databricks on AWS preferred; Azure Databricks experience is also acceptable.
• Strong proficiency in Spark, including PySpark and/or Scala, and SQL.
• Proven experience participating in the migration of on-premises SQL Server data warehouses to cloud-based data platforms.
• Experience converting SSIS-based ETL pipelines into Spark-based data engineering solutions.
• Experience working within structured migration or enterprise modernization initiatives rather than solely lift-and-shift projects.
• Strong understanding of data warehousing concepts, dimensional modeling, and analytical workloads.
• Experience with Delta Lake, incremental processing patterns, and data versioning.
• Familiarity with Databricks Workflows, Jobs, and production-grade deployments.
• Practical experience with performance tuning and large-volume data processing.

Preferred Skills

• Experience modernizing SSAS cube-based reporting solutions into Lakehouse-native semantic models.
• Exposure to Databricks SQL Warehouses and BI integrations, with Power BI experience preferred.
• Working knowledge of Medallion architecture and cloud-native data engineering practices aligned with Databricks best practices.
• Familiarity with MLflow, feature engineering, feature stores, or other AI/ML capabilities within Databricks.
• Databricks certification is a plus.
• Location: Pune, Maharashtra
• Employment Type: Direct Hire or Contract-to-Hire
• Work Arrangement: Hybrid, with a minimum of three days per week in the Pune office, preferably Tuesday, Wednesday, and Thursday

Additional Information

Customized Energy Solutions India offers competitive compensation along with performance-based incentives tied to individual and company performance. Our benefits are designed to support both professional and personal well-being and include comprehensive medical insurance coverage for associates and eligible dependents, generous paid time off, and opportunities for professional development through training and skill-building programs.

We are committed to creating an environment where associates are supported, challenged, and provided opportunities to grow and develop their careers over the long term.