Python PySpark Developer

Tata Consultancy Services; Chennai District, Bengaluru, Pune District; On-site; INR 600,000 - 1,200,000; Full time

Job summary

Tata Consultancy Services is seeking a skilled data engineer based in Chennai, Tamil Nadu. The role involves implementing data ingestion pipelines, working with both structured and unstructured data, and developing scalable cloud-based solutions using PySpark.

The ideal candidate should have experience in ETL processes and data warehouse transformation. A strong understanding of data quality and performance optimization is essential. This position offers the chance to apply best practices in automation and testing to improve data processing frameworks.

Qualifications

• Experience in implementing data ingestion pipelines from various sources.
• Proficiency in building ETL/Data Warehouse transformation processes.
• Experience with both structured and unstructured data.

Skills

Data ingestion pipeline implementation

ETL/Data Warehouse experience

Working with structured and unstructured data

Big Data solutions in PySpark

Scalable frameworks for data processing

Data quality and consistency

Performance analysis and optimization

Best practices in Automation and Testing

Job description

Must have Skills:

• Implementing data ingestion pipelines from different types of data sources i.e Databases, S3, Files etc..
• Experience in building ETL/ Data Warehouse transformation process.
• Experience working with structured and unstructured data.
• Developing Big Data and non-Big Data cloud-based enterprise solutions in PySpark and SparkSQL and related frameworks/libraries.
• Developing scalable and re-usable, self-service frameworks for data ingestion and processing.
• Integrating end to end data pipelines to take data from data source to target data repositories ensuring the quality and consistency of data.
• Processing performance analysis and optimization.
• Bringing best practices in following areas: Design & Analysis, Automation (Pipelining, IaC), Testing, Monitoring, Documentation.

Good to have (Knowledge):

• Experience in cloud-based solutions,
• Knowledge of data management principles