Senior Data Engineer

Recro; Bengaluru; On-site; INR 1,600,000 - 2,000,000; Full time

Job summary

Recro is seeking a Senior Data Engineer to develop a long-term vision for a scalable data platform and robust Data Ops practices. You will design data flows and governance in cloud or Hadoop environments, automate provisioning, testing, and monitoring to shorten development cycles and boost deployment frequency.

You will lead data engineering workstreams, working with Python/PySpark, SQL, and ETL frameworks like Airflow, Oozie, and Jenkins, to build production-grade pipelines and ensure data

Qualifications

• Bachelors or Masters in computer science, information systems or equivalent field.
• 5+ years building data flows and data management on modern big data tech stack.

Responsibilities

• Develop long-term vision for a scalable data platform and Data Ops practices.
• Design and architect data flows and data management in Hadoop or cloud environments.
• Promote Data Ops to automate data provisioning, testing and monitoring.
• Establish governance, CI/CD and test coverage for mature data pipelines.
• Evaluate tools/tech strategy for analytics platforms with Enterprise Architecture.
• Lead data engineering workstreams with a product mindset.

Skills

Big data engineering

Data pipelines

ETL development

Airflow

Python

PySpark

SQL

Unix scripting

Education

Bachelor's or Master's in CS/IS or equivalent

Tools

Airflow

Oozie

Jenkins

Job description

Develop long-term vision for a highly scalable data platform, data management and Data Ops practices.

Design and architect data flows, data management in Hadoop or Cloud environment which are scalable, repeatable and eliminate time consuming steps Promote Data Ops approach to automate the provision of data, testing and monitoring and to shorten development cycles and increase deployment frequency.

Establish development and data governance processes to build mature data pipelines, CI/CD, test coverages, etc.

Evaluate, provide insights and recommendations on tools and technology strategy for analytics data platforms and applications in conjunction with Enterprise Architecture team

Ability to lead data engineering workstreams with a product mindset

Who are we looking for?

Bachelor's or master's degree in computer science, Information Systems or equivalent field.

At least 5+ years of experience in building data flows and data management on modern big data tech stack Data Strategy: Understands, articulates, and applies principles of the defined strategy to routine business problems that involve a single function.

Data Transformation and Integration: Extracts data from identified databases. Creates data pipelines and transform data to a structure that is relevant to the problem by selecting appropriate techniques.

Develops knowledge of current analytics trends.

Data Source Identification: Supports the understanding of the priority order of requirements and service level agreements.

Helps identify the most suitable source for data that is fit for purpose.

Demonstrates expertise in writing complex, highly optimized queries across large data sets Strong experience in using ETL framework (eg. Airflow, Oozie, Jenkins etc.) to build and deploy production-quality ETL pipelines.

Experience in ingesting and transforming structured and unstructured data from internal and third-party sources into dimensional models.

Knowledge of data structures and distributed computing.

Should be comfortable in manipulation and analysis of high-volume data from variety of internal and third-party sources.

Experience in one or more programming languages like Python or PySpark and moderate knowledge on unix scripting.