Data Engineer

Cynosure Corporate Solutions; Chennai District; On-site; INR 800,000 - 1,800,000; Full time

Job summary

Cynosure Corporate Solutions in Chennai is seeking a data engineer to design and implement data pipelines for the Data Science team. You will participate in system design meetings, gather requirements, and convert them into scalable ETL solutions using PySpark, Spark, SCALA, and SQL.

The role requires hands-on experience with Python, Snowflake, cloud platforms (Azure/AWS), and knowledge of Hive/Impala/Hadoop.

Qualifications

• Degree in Computer Science or IT; Master’s is a plus.
• Hands-on Python and PySpark experience.
• Hands-on Spark and SCALA experience.
• Hands-on Snowflake experience.
• Strong numerical and analytical skills.
• Experience with cloud platforms such as MS Azure, AWS, etc.
• Knowledge of data models, data mining, and segmentation techniques.

Responsibilities

• Participate in customer system design meetings to gather requirements.
• Build data pipelines for Data Science teams.
• Work with ETL processes and tools.
• Develop Python and PySpark/Scala solutions with Hive, Impala, Hadoop and SQL.
• Validate SQL, tune queries and optimize performance.
• Guide QA in defining system integration tests.
• Split deliverables into tasks and assign to team.
• Deploy ETL code and follow Agile methodology.

Skills

Python

PySpark

Spark

SCALA

SQL

SQL Tuning

Shell Scripting

Data Pipelines

Cloud Platforms

Kubernetes

CI/CD

Apache Kafka

Data Integration

QA Guidance

Education

Bachelor’s in Computer Science/IT

Master’s degree in CS/IT

Tools

Snowflake

Hive

Impala

Airflow

Job description

Cynosure Corporate Solutions | Full time

Cynosure Corporate Solutions is a human capital services company. Our focus is to provide Executive Search, Recruitment, Training, Temporary Staffing services, Statutory Compliance's, and other HR aligned services. We understand the business goals of our clients and their need to align human resources to these goals. We are committed to provide high quality manpower in accordance to global standards and their requirements. Our candidates stand testimony to our professional competency.

We work with clients in IT/ITES, Manufacturing, Automobile, Health, Telecom, Media.

Job Description

• Participate in the customer’s system design meetings and collect the functional/technical equirements.
  
• Build up data pipelines for consumption by the Data Science team.
  
• Experience with ETL process and tools.
  
• Clear understanding and experience with Python and PySpark or Spark and SCALA, with HIVE, irflow, Impala, and Hadoop and RDBMS architecture.
  
• Experience in writing Python programs and SQL queries.
  
• Experience in SQL Query tuning.
  
• Experienced in Shell Scripting (Unix/Linux).
  
• Build and maintain data pipelines in Spark/PySpark with SQL and Python or SCALA.
  
• Knowledge of Cloud (Azure/AWS/GCP, etc..) technologies.
  
• Good to have knowledge of Kubernetes, CI/CD concepts, Apache Kafka.
  
• Suggest and implement best practices in data integration.
  
• Guide the QA team in defining system integration tests as needed.
  
• Split the planned deliverables into tasks and assign them to the team.
  
• Needs to Maintain/Deploy the ETL code and follow the Agile methodology.
  
• Needs to work on optimization wherever applicable.
  
• Good oral, written and presentation skills.
  

Requirements

• Degree in Computer Science, IT, or a similar field; a Master’s is a plus.
  
• Hands-on experience with Python and PySpark.
  
• Hands-on experience with Spark and SCALA.
  
• Hands-on experience with Snowflake
  
• Great numerical and analytical skills.
  
• Working knowledge of cloud platforms such as MS Azure, AWS, etc.
  
• Technical expertise in data models, data mining, and segmentation techniques.