Digital : Python ,Pyspark, GCP

Tata Consultancy Services; Hyderabad; On-site; INR 1,800,000 - 2,400,000; Full time

Job summary

Tata Consultancy Services in Hyderabad is seeking a senior data engineer with deep expertise in big data ecosystems to design and implement robust data pipelines. The role requires hands-on Spark, PySpark, Python, and Scala, with experience in data lakes, Delta Lake, and cloud services.

You will also work with CI/CD tools and orchestration frameworks to ensure scalable, secure data solutions. Strong knowledge of functional programming concepts, Cats/Shapeless, SQL tuning, and

Qualifications

• 6 years of total experience in data engineering or big data development.
• 2-3 years hands-on experience with Apache Spark.
• Strong programming skills in PySpark, Python, and Scala.
• 2+ years of experience in Scala backend development.
• Proficient in Scala, both object oriented and functional programming concepts.
• Hands-On knowledge with Scala Typelevel libraries like Cats, Shapeless, and others used for building applications with strong typing and efficient concurrency.
• Solid understanding of data lakes, lakehouses, and Delta Lake concepts.
• Experience in SQL development and performance tuning.
• Proficient in cloud services (e.g. S3, Glue, Lambda, EMR, Redshift, CloudWatch, IAM).
• Familiarity with Airflow, dbt, or similar orchestration tools is a plus.
• Experience in CI/CD tools like Jenkins, GitHub Actions, or Code Pipeline.
• Knowledge of data security, governance, and compliance frameworks.

Skills

Spark

PySpark

Python

Scala

Scala backend

Functional programming

Cats/Shapeless

Data lakes/Delta Lake

SQL

Cloud services

Airflow/dbt

CI/CD

Security/Governance

Big data development

Tools

Jenkins

GitHub Actions

CodePipeline

Airflow

dbt

Job description

Role & responsibilities

• 6 years of total experience in data engineering or big data development.
• 2-3 years hands-on experience with Apache Spark.
• Strong programming skills in PySpark, Python, and Scala.
• 2+ years of experience in Scala backend development.
• Proficient in Scala, both object oriented and functional programming concepts.
• Deep understanding and application of advanced functional programming concepts like category theory, monads, applicatives, and type classes.
• Hands-On knowledge with Scala Typelevel libraries like Cats, Shapeless, and others used for building applications with strong typing and efficient concurrency.
• Solid understanding of data lakes, lakehouses, and Delta Lake concepts.
• Experience in SQL development and performance tuning.
• Proficient in cloud services (e.g. S3, Glue, Lambda, EMR, Redshift, CloudWatch, IAM).
• Familiarity with Airflow, dbt, or similar orchestration tools is a plus.
• Experience in CI/CD tools like Jenkins, GitHub Actions, or Code Pipeline.
• Knowledge of data security, governance, and compliance frameworks.