Data Engineer

Samta - 2023; Nagpur District; On-site; INR 1,200,000 - 2,000,000; Full time

Job summary

A leading technology company is seeking a Senior Data Engineer located in Maharashtra, India. The successful candidate will have expertise in data platforms like Snowflake and Databricks, along with a strong understanding of machine learning and data governance practices. This role involves designing scalable data models, data migration, and ensuring data security and compliance. Applicants should have advanced SQL skills and proficiency in Python. Competitive compensation is offered.

Qualifications

• Hands-on experience with data management products like Microsoft Fabric, Snowflake, or Databricks.
• Strong skills in performance tuning and optimization.
• Proficiency with ML lifecycle management and data governance.
• Knowledge of ML concepts and ML lifecycle tools.

Responsibilities

• Design and develop scalable data models.
• Migrate data from various sources and develop ETL pipelines.
• Implement data security, compliance, and governance policies.
• Implement data governance policies within solutions.
• Prototype ML preprocessing, feature engineering and evaluation.
• Handle ML lifecycle and MLOps practices using tools.

Skills

Data Modeling

Data Governance

Machine Learning

Performance Optimization

Data Migration

MLOps Practices

Data Storage & Lakes

Containerization

Database Design & Performance

Tools

Snowflake

Databricks

SQL

Python

Docker

Apache Airflow

TensorFlow

PyTorch

Scikit-learn

Delta Lake

Docker

Vertex AI

AWS SageMaker

Azure ML

Job description

As a Senior Data Engineer, this position requires deeper skills and hands-on experience on oneof the Data management products such as Microsoft Fabric, Snowflake or Databricks. You areexpected to assist our clients in conceptualising, implementation and management of Dataplatforms. Furthermore, it is important to have competencies in Data models and relatedarchitectures, visualisation, governance, training AI models and tuning AI applications.Knowledge of creating synthetic data is preferred.

Job Description

• Data Modeling & Architecture: Expertise in designing and developing scalable data models in Snowflake/Databricks, including relational and dimensional principles.
• Data Migration & Pipelines: Proven experience in data platform transformation, migrating data from various sources to Snowflake/Databricks and developing efficient ETL pipelines using SQL, Python, and ETL tools.
• Performance Optimization: Strong skills in performance tuning and optimization for data models, queries, and pipelines, including partitioning and data compression.
• Data Governance: Knowledge of implementing data security, compliance, and governance policies within data solutions.
• Machine Learning (ML): Solid understanding of ML and deep learning algorithms, including data preprocessing, feature engineering, and model evaluation using frameworks like TensorFlow, PyTorch, and Scikit-learn.
• MLOps: Hands-on experience with ML lifecycle management, experiment tracking, and workflow automation using tools like MLflow and Apache Airflow. Experience with MLOps platforms such as AWS SageMaker, Azure ML, Google Vertex AI, or Databricks is a plus.
• Data Storage: Practical experience implementing and managing data lakes with Delta Lake.
• Containerization: Proficiency with Docker for building portable data and ML solutions
• Database Management: Ability to design database objects, monitor performance, and provide technical support.
• SQL Proficiency: Advanced SQL skills for data manipulation and querying in Snowflake/Databricks.