Sr. Software Engineer - Databricks

Yash Technologies; Pune District; On-site; INR 2,500,000 - 4,000,000; Full time

Job summary

Yash Technologies in India is seeking Databricks professionals for a Data Engineer role, focusing on Databricks, PySpark, Python, and SQL to build scalable data solutions. Experience 4-6 years is preferred.

You will design and implement ETL/ELT pipelines, migrate DB2 components to AWS PostgreSQL, and ensure data quality, governance and security during migration projects. Collaboration with cross-functional teams is essential.

Qualifications

• Strong SQL skills with stored procedures, functions and query optimization.
• Hands-on PySpark, Python and Databricks for scalable data solutions.
• Experience with ETL/ELT pipelines for data ingestion, transformation and integration.
• Experience with AWS data services such as RDS, S3 and Glue.
• PostgreSQL design, querying, performance tuning and data modeling.
• Data migration, integration and data products development.
• Knowledge of Databricks workflow orchestration and monitoring.
• Data cleansing, validation, reconciliation and quality management.
• Strong analytical and problem-solving abilities.
• Understanding of data warehousing concepts and CI/CD practices.

Responsibilities

• Analyze key mainframe jobs and identify DB2 components for migration.
• Assess DB2 schemas, tables and dependent objects for migration.
• Lead decommissioning of DB2 schemas as part of modernization.
• Plan and execute migration of identified DB2 schemas and data to AWS PostgreSQL.
• Perform schema conversion, data validation and post-migration testing.
• Migrate schemas/data to AMS while ensuring data integrity and business continuity.
• Collaborate with cross-functional teams for migration planning, cutover activities and production deployment.
• Ensure compliance with data quality, governance and security standards throughout migration.

Skills

Analytical thinking

Problem solving

Communication

Independent work

Team collaboration

Tools

Databricks

PySpark

Python

SQL

PostgreSQL

ETL/ELT

AWS (S3, RDS, Glue)

Data Migration

Scala

Git

CI/CD

Job description

We are looking forward to hire Databricks Professionals in the following areas :

Data Engineer -- Databricks, Python, ETL, Postgres

Experience Level 4 -6 Years

:

• Strong expertise in SQL , including stored procedures, functions, indexes, query optimization, and performance tuning.
• Hands-on experience with PySpark, Python, and Databricks for designing and developing scalable data solutions.
• Strong experience in designing, developing, and supporting ETL/ELT pipelines for data ingestion, transformation, and integration.
• Experience working with AWS cloud data services such as RDS, S3, and Glue , and building end-to-end data pipelines.
• Strong experience with PostgreSQL , including database design, query development, performance tuning, data modeling, and database administration concepts.
• Experience in data migration , data integration, organization-wide reporting, and building data products.
• Good understanding of Databricks workflow orchestration, pipeline execution, monitoring, and optimization .
• Experience in data cleansing, data validation, reconciliation, and data quality management .
• Strong analytical and problem-solving skills with the ability to organize, analyze, and interpret large datasets.
• Good understanding of data warehousing concepts and best practices.
• Knowledge of Scala, Git, and CI/CD practices for code versioning and deployment automation.
• Strong verbal, written, and interpersonal communication skills.
• Organized and structured approach with the ability to work independently and collaboratively in a team environment.

Migration Modernization Responsibilities

• Analyze key mainframe jobs and identify DB2 database components requiring migration.
• Identify and assess DB2 schemas, tables, and dependent objects for migration.
• Lead the decommissioning of DB2 schemas as part of application and database modernization initiatives.
• Plan and execute migration of identified DB2 schemas and data to AWS PostgreSQL .
• Perform schema conversion, data validation, reconciliation, and post-migration testing.
• Migrate identified schemas/data to AMS while ensuring data integrity and business continuity.
• Collaborate with cross-functional teams to support migration planning, cutover activities, and production deployment.
• Ensure compliance with data quality, governance, and security standards throughout the migration lifecycle.

Primary Skills

Databricks, PySpark, Python, SQL, PostgreSQL, ETL/ELT, AWS (S3, RDS, Glue), Data Migration

Secondary Skills

Scala, Git, CI/CD, DB2 Migration, Mainframe Modernization, Data Quality, Data Validation Reconciliation

Our Hyperlearning workplace is grounded upon four principles

• Flexible work arrangements, Free spirit, and emotional positivity
• Agile self-determination, trust, transparency, and open collaboration
• All Support needed for the realization of business goals,
• Stable employment with a great atmosphere and ethical corporate culture