Data Engineer

Inference Labs; Bengaluru; On-site; INR 1,200,000 - 2,200,000; Full time

Job summary

Inference Labs in Bengaluru is seeking a Data Engineer to design, develop, and maintain ETL pipelines and manage database systems. The ideal candidate will have 4-7 years of experience and strong SQL, data modeling, and ETL proficiency.

You'll profile source data, ensure data quality, build scalable data stores, and implement data pipelines with Spark, Python, and PySpark across cloud environments such as Azure and Databricks.

Qualifications

• 4-7 years of experience in data engineering
• Strong SQL, database design and ETL experience
• Proficient in Python and PySpark
• Experience with Spark clusters and Databricks
• Experience with relational databases (PostgreSQL, MySQL, Oracle) and NoSQL (MongoDB, Cassandra, DynamoDB)
• Knowledge of data modeling (star/snowflake, data vault) and semantic modelling
• Experience designing and maintaining ETL pipelines and data warehouses
• Familiarity with cloud platforms (Azure, AWS, GCP)
• Excellent data quality and data profiling skills
• Strong problem-solving and communication skills

Responsibilities

• Analyse the different source systems, profile data, understand, document & fix Data Quality issues
• Gather requirements and business process knowledge to transform the data in a way that is geared towards the needs of end users
• Write complex SQLs to extract & format source data for ETL/data pipeline
• Design, implement, and maintain systems that collect and analyze business intelligence data.
• Design and architect an analytical data store or cluster for the enterprise and implement data pipelines that extract, transform, and load data into an information product that helps the organization reach strategic goals.
• Create design documents, Source to Target Mapping documents and any supporting documents needed for deployment/migration
• Design, Develop and Test ETL/Data pipelines
• Design & build metadata-based frameworks needs for data pipelines
• Write Unit Test cases, execute Unit Testing and document Unit Test results
• Manage and maintain the database, warehouse, & cluster with other dependent infrastructure.
• Expertise in managing and optimizing Spark clusters, along with other implementations of Spark.
• Strong programming skills in Python and Py-spark.
• Strong proficiency in SQL and experience with relational databases (PostgreSQL, MySQL, Oracle, etc.) and NoSQL databases (MongoDB, Cassandra, DynamoDB).
• Knowledge of data modelling techniques such as star/snowflake, data vault, etc.
• Knowledge of semantic modelling
• Strong problem-solving skills - Be able to hone business acumen with a capacity for straddling between macro business strategy to micro tangible data and AI products.
• Perform data cleaning, transformation, and validation to ensure accuracy and consistency across various data sources
• Technologies preferred – Azure, Databricks

Skills

SQL

Python

PySpark

Data modeling

Data quality

Big data

Education

B.E./B.Tech in any specialization

BCA

M.Tech in any specialization

MCA

Tools

Databricks

Azure

PostgreSQL

MySQL

Oracle

MongoDB

Cassandra

DynamoDB

Spark

AWS

GCP

Job description

Job Experience:4-7 years

We are seeking a Data Engineer to join our growing team. The Data Engineer will be responsible for designing, developing, and maintaining our ETL pipelines and managing our database systems. The ideal candidate should have a strong background in SQL, database design, and ETL processes.

Responsibilities for the job

Key Responsibilities: –

• Analyse the different source systems, profile data, understand, document & fix Data Quality issues

• Gather requirements and business process knowledge to transform the data in a way that is geared towards the needs of end users

• Write complex SQLs to extract & format source data for ETL/data pipeline

• Design, implement, and maintain systems that collect and analyze business intelligence data.

• Design and architect an analytical data store or cluster for the enterprise and implement data pipelines that extract, transform, and load data into an information product that helps the organization reach strategic goals.

• Create design documents, Source to Target Mapping documents and any supporting documents needed for deployment/migration

• Design, Develop and Test ETL/Data pipelines

• Design & build metadata-based frameworks needs for data pipelines

• Write Unit Test cases, execute Unit Testing and document Unit Test results

• Manage and maintain the database, warehouse, & cluster with other dependent infrastructure.

• Expertise in managing and optimizing Spark clusters, along with other implementations of Spark.

• Strong programming skills in Python and Py-spark.

• Strong proficiency in SQL and experience with relational databases (PostgreSQL, MySQL, Oracle, etc.) and NoSQL databases (MongoDB, Cassandra, DynamoDB).

• Knowledge of data modelling techniques such as star/snowflake, data vault, etc.

• Knowledge of semantic modelling

• Strong problem-solving skills - Be able to hone business acumen with a capacity for straddling between macro business strategy to micro tangible data and AI products.

• Perform data cleaning, transformation, and validation to ensure accuracy and consistency across various data sources

• Technologies preferred – Azure, Databricks

Eligibility Criteria for the Job

Education: -

B.E/B.Tech in any specialization, BCA, MTech in any specialization, MCA.

Primary Skill: -

1. SQL
2. Databricks
3. Any one of the cloud experiences (AWS, Azure, GCP).

4. Python, Pyspark

Management Skills: -

1. Ability to handle given tasks and projects simultaneously in an organized and timely manner.

Soft Skills: -

1. Good communication skills, verbal and written.
2. Attention to details.
3. Positive attitude and confidence.