Senior / Lead Data Engineer – Databricks, PySpark & Applied AI/LLM Engineering

Srijan Technologies PVT LTD; Gurugram District; On-site; INR 1,500,000 - 2,500,000; Full time

Benefits offered by this job

Professional Development and Mentorship

Health and Family Insurance

40+ Leaves per year

Wellness and Counselling sessions

Job summary

Srijan Technologies PVT LTD in Gurugram seeks a Senior Data Engineer to optimize and implement data architecture for retail operations. You will design and build data models, develop ETL processes, and ensure data integrity across platforms. The ideal candidate has over 5 years' experience in data engineering with proficient skills in PySpark and Databricks. The company offers a hybrid work model, including health insurance and extensive paid leave.

Qualifications

• 5+ years of experience in data engineering, preferably in retail.
• Proven experience as a senior Data Engineer or Lead.
• Strong programming skills in PySpark and optimization.

Responsibilities

• Design and develop data models and pipelines for retail data.
• Hands-on Spark programming for data processing applications.
• Collaborate with DevOps to maintain CI/CD pipelines.

Skills

PySpark programming

Data modeling

Data integration

ETL/ELT processes

Data governance

Optimization techniques

Communication skills

Problem-solving abilities

Tools

Databricks

Delta Lake

GitLab

Cloud services (AWS/Azure/GCP)

Job description

JD_Senior Data Engineer

Senior/Lead Data Engineer - Databricks We are seeking a highly skilled and experienced Data Engineering Lead with a strong background in the retail domain and exceptional programming abilities. As a Lead, you will play a pivotal role in implementing, and optimizing data architecture to support our retail business operations and analytics initiatives. Your expertise in Spark programming, optimization techniques, and familiarity with Databricks and CI/CD practices will be instrumental in ensuring the efficient and effective management of our data ecosystem.

Responsibilities

• Design and develop data models, data integration processes, and data pipelines to capture, transform, and load structured and unstructured data from various retail sources.
• Hands‑on programming in Spark to develop and optimize data processing applications and analytics workflows.
• Apply optimization techniques to enhance the performance and efficiency of data processing and analytical tasks.
• Evaluate and implement appropriate tools and technologies, including Databricks, to streamline data operations and ensure scalability and reliability.
• Work closely with other team members to ensure data integrity, consistency, and accessibility across the organization.
• Define and enforce best practices for data governance and data management, including data quality, metadata management, and data security.
• Collaborate with DevOps teams to establish and maintain CI/CD pipelines for data engineering and analytics workflows.
• Peer review of team members' deliverables.
• Stay updated with the latest advancements and trends in the retail domain, data architecture, and programming languages to drive continuous improvement.

Requirements

• At least 5+ years of experience in the data engineering domain.
• Proven experience as a senior Data Engineer /Lead, preferably within the retail industry.
• Strong programming skills with expertise in PySpark programming and optimization techniques.
• Hands‑on experience with Databricks, Delta Lake and its components for data processing and analytics.
• Hands‑on experience in data modelling, data integration, and ETL/ELT processes.
• Experience working with GitLab pipelines and an in-depth understanding of CI/CD pipeline designs.
• Experience with data governance, data quality, and metadata management.
• Strong analytical and problem‑solving abilities with a detail‑oriented mindset.
• Excellent communication and collaboration skills to work effectively with cross‑functional teams.
• Ability to adapt to a fast‑paced and evolving environment while managing multiple priorities.
• Good to have experience in at least one of the Cloud Vendor (AWS / Azure / GCP).
• Good to have experience with streaming technologies.

What We Offer

• Professional Development and Mentorship.
• Hybrid work mode with remote friendly workplace. (6 times in a row Great Place To Work Certified).
• Health and Family Insurance.
• 40+ Leaves per year along with maternity & paternity leaves.
• Wellness, meditation and Counselling sessions.