Data Engineer

wk; India; On-site; INR 1,800,000 - 3,000,000; Full time

Job summary

wk is seeking a Senior Product Software Data Engineer to design and build scalable data pipelines and real-time analytics capabilities. You will collaborate with cross-functional teams to translate business needs into robust data solutions and contribute to the ongoing improvement of our products.

You will work with Scala, Spark, and Azure Databricks, partner with data scientists, and drive data governance and quality across platforms. Strong communication and mentoring skills are valued.

Qualifications

• 3+ years of experience in data engineering.
• 2+ years building data pipelines for structured and unstructured data.
• 1+ years with big data tools including PySpark, Hadoop, Kafka, EventHub, and Stream Analytics.
• 1+ years of experience in relational and dimensional data modeling using SQL Server and data warehousing concepts.
• Must have 2+ years of experience with T-SQL, stored procedures, and performance tuning.
• 2+ years of experience with Azure Data Factory, Azure Data Lake, Azure SQL DB, and Azure Synapse Analytics.
• Good to have 2+ years of experience with Azure ML, Python, and Scala.
• Good to have 2+ years of experience with Power BI and Azure Analysis Services.

Responsibilities

• Collaborate with cross-functional teams to review use cases and translate business requirements into scalable data solutions.
• Design and develop highly optimized, reliable, and secure enterprise-level data pipelines for real-time and batch processing.
• Analyze existing business processes, identify gaps, and recommend data-driven improvements.
• Build modern real-time data solutions using Scala, Apache Spark, and Azure Databricks.
• Partner with data scientists and business stakeholders to operationalize machine learning models.
• Champion data governance, metadata management, and operational excellence across data platforms.
• Ensure data quality and integrity by developing validation frameworks and monitoring mechanisms.
• Adhere to coding standards, participate in peer code reviews, and contribute to technical documentation.
• Mentor junior engineers and contribute to knowledge sharing across the team.

Skills

Software Engineering

Software Development

Problem Solving

Analysis

Testing

Agile

Source Code Repository

Relational Database

Documentation

Education

Bachelor's degree in CS/Engineering

Tools

PySpark

Hadoop

Kafka

EventHub

Azure Databricks

SQL Server

Azure Synapse

Job description

About the Role :

Take your software engineering career to the next level as a Senior Product Software Data Engineer. You will play a key role in designing, developing, and troubleshooting software programs. Your technical skills and innovative mindset will contribute to the continuous improvement of our products, making a significant impact on user experience and satisfaction.

Responsibilities :

• Collaborate with cross-functional teams to review use cases and translate business requirements into scalable data solutions.
• Design and develop highly optimized, reliable, and secure enterprise-level data pipelines for real-time and batch processing.
• Analyze existing business processes, identify gaps, and recommend data-driven improvements.
• Build modern real-time data solutions using Scala , Apache Spark , and Azure Databricks .
• Partner with data scientists and business stakeholders to operationalize machine learning models.
• Champion data governance, metadata management, and operational excellence across data platforms.
• Ensure data quality and integrity by developing validation frameworks and monitoring mechanisms.
• Adhere to coding standards, participate in peer code reviews, and contribute to technical documentation.
• Mentor junior engineers and contribute to knowledge sharing across the team.

Skills :

• - Software Engineering: The ability to design, develop, and maintain software systems and applications by applying principles and techniques of computer science, engineering, and mathematical analysis. This includes the capacity to understand user requirements, create and test the software, and resolve any software-related issues.
• - Software Development: The ability to design, write, test, and implement software programs, applications, and systems. This includes understanding various programming languages, software architecture, and software testing methods. It also involves problem- solving capabilities to fix software issues and improve functionality.
• - Problem Solving: The ability to understand a complex situation or issue and devise a solution by defining the problem, identifying potential strategies, and ultimately choosing and implementing the most effective course of action.
• - Analysis: The ability to examine complex situations or problems, break them down into smaller parts, and understand how these parts work together.
• - Testing: The skill of evaluating a system or process, often in software or product development. It involves the ability to identify problems, measure effectiveness, and ensure quality or functionality.
• - Agile: The ability to swiftly and effectively respond to changes, with an emphasis on continuous improvement and flexibility. In the context of project management, it denotes a methodology that promotes adaptive planning and encourages rapid and flexible responses to changes.
• - Source Code Repository: The ability to effectively use a source code repository, a file archive and web hosting facility where a large amount of source code, for software or for web pages, is kept, either publicly or privately. This skill involves the ability to manage and track changes to code, identify and fix issues, merge code from different branches, and collaborate with other developers.
• - Relational Database: The ability to design, implement, and manipulate a relational database, a type of database that stores and organizes data in a structured way and where data is logically interrelated. This skill often requires proficiency in SQL, database management systems, and understanding of database design principles.
• - Documentation: The ability to create, manage, organize, and maintain important information and documents in various physical and digital formats. This skill may include preparing reports, managing files, storing data, and keeping records organized and updated for easy retrieval and understanding.

Experience and Qualifications

• 3 + years of overall experience in data engineering or related fields.
• 2 + years of experience building data pipelines for structured and unstructured data.
• 1 + years of experience with big data technologies including PySpark , Hadoop , Kafka , EventHub , and Stream Analytics .
• 1 + years of experience in relational and dimensional data modeling using SQL Server and data warehousing concepts.
• Must have 2 + years of experience with T-SQL , stored procedures, and performance tuning.
• 2 + years of experience with Azure Data Factory , Azure Data Lake , Azure SQL DB , and Azure Synapse Analytics .
• Good to have 2+ years of experience with Azure ML , Python , and Scala .
• Good to have 2+ years of experience with Power BI and Azure Analysis Services .
• Experience working with business stakeholders for requirements gathering and use case analysis.
• Strong communication, collaboration, and creative problem-solving skills.

Preferred Qualifications

• Bachelor's degree in computer science, Engineering, or equivalent practical experience.
• Experience with Agile/Scrum methodologies and DevOps practices.
• Fam