Senior Data Engineer

Mags Consulting; Chennai; On-site; INR 1,200,000 - 2,200,000; Full time

Job summary

Mags Consulting in Chennai, Tamil Nadu, is seeking a data engineer to design, develop, and maintain scalable data pipelines. You will build distributed data processing solutions using Apache Spark and PySpark, ingest and transform data from multiple sources, and work with AWS data engineering services.

You will optimize SQL queries, manage relational and non-relational databases, implement data quality checks, troubleshoot issues, and collaborate with stakeholders to deliver robust data

Qualifications

• Strong/advanced SQL with complex queries and optimization.
• Experience with Spark and distributed data processing.
• Data ingestion, transformation and integration from multiple sources.
• Experience with JSON, logs, PDFs, and other formats.

Responsibilities

• Design, develop, and maintain scalable and high-performance data pipelines.
• Build distributed data processing solutions using Apache Spark / PySpark.
• Develop data ingestion, transformation, and integration processes from multiple data sources.
• Work with AWS data engineering and cloud processing services.
• Process structured, semi-structured, and unstructured data including JSON, logs, PDFs, and other file formats.
• Develop complex and optimized SQL queries for data processing and analysis.
• Work with both relational and non-relational databases.
• Implement data validation, quality checks, and monitoring processes.
• Troubleshoot complex data issues and perform root-cause analysis.
• Optimize data pipelines, queries, and distributed processing jobs for performance and scalability.
• Collaborate with stakeholders to understand business requirements and translate them into scalable data solutions.
• Take ownership of data engineering initiatives and deliver solutions independently.
• Stay updated with emerging AWS, Big Data, and Data Engineering technologies and best practices.

Skills

SQL

Apache Spark

PySpark

AWS

Data ingestion

Data transformation

JSON

PDFs

Tools

Distributed processing

Job description

• Design, develop, and maintain scalable and high-performance data pipelines .
  
• Build distributed data processing solutions using Apache Spark / PySpark .
  
• Develop data ingestion, transformation, and integration processes from multiple data sources.
  
• Work with AWS data engineering and cloud processing services .
  
• Process structured, semi-structured, and unstructured data including JSON, logs, PDFs, and other file formats .
  
• Develop complex and optimized SQL queries for data processing and analysis.
  
• Work with both relational and non-relational databases .
  
• Implement data validation, quality checks, and monitoring processes.
  
• Troubleshoot complex data issues and perform root-cause analysis.
  
• Optimize data pipelines, queries, and distributed processing jobs for performance and scalability.
  
• Collaborate with stakeholders to understand business requirements and translate them into scalable data solutions.
  
• Take ownership of data engineering initiatives and deliver solutions independently.
  
• Stay updated with emerging AWS, Big Data, and Data Engineering technologies and best practices .
  

Requirements

• Strong/advanced SQL, including complex queries and optimization
  
• Apache Spark and distributed data processing
  
• Data ingestion, transformation and integration
  
• Structured, semi-structured and unstructured data — JSON, logs, PDFs, etc.
  
• Relational and non-relational databases
  
• Data quality, validation, performance tuning and optimization
  
• Deep-dive data analysis and troubleshooting