Senior Databricks Python Data Engineer

Tata Consultancy Services; Jersey City (NJ); On-site; USD 100,000 - 115,000; Full time

Job summary

Tata Consultancy Services in Jersey City is seeking a senior Data Engineer to design, develop, and maintain data pipelines using Databricks, Python, and PySpark. Collaborate with data architects, analysts, and business teams to deliver scalable data solutions for large volumes of structured and unstructured data.

You will optimize batch and streaming workflows, ensure data quality, governance, and security, and participate in code reviews and Agile ceremonies to drive continuous improvement.

Qualifications

• 10+ years of experience in Data Engineering, Data Warehousing, or Big Data development.
• 5+ years of hands-on experience with Databricks, Python, and PySpark.
• Experience building and supporting large-scale ETL/ELT data pipelines.
• Strong knowledge of Spark SQL, Delta Lake, and Data Lakehouse architecture.
• Experience with data modeling, data transformation, and data quality processes.
• Knowledge of orchestration tools such as Airflow, ADF, or similar.
• Experience with Git, CI/CD, and Agile development methodologies.
• Strong analytical, debugging, and performance tuning skills.
• Excellent verbal and written communication skills.

Responsibilities

• Design, develop, and maintain data pipelines using Databricks, Python, and PySpark.
• Build scalable data ingestion, transformation, and integration solutions.
• Collaborate with Data Architects, Analysts, and business teams to deliver data solutions.
• Develop and optimize batch and real-time data processing workflows.
• Ensure data quality, integrity, security, and governance standards are met.
• Monitor, troubleshoot, and resolve production data pipeline issues.
• Optimize Spark jobs, SQL queries, and Databricks workloads for performance.
• Participate in data modeling, code reviews, and platform enhancement activities.
• Create technical documentation and support production deployments.
• Participate in Agile ceremonies and contribute to continuous improvement initiatives.

Skills

Data engineering

Data warehousing

Big data development

SQL skills

Analytical thinking

Communication skills

Agile methodologies

Tools

Databricks

Python

PySpark

Spark SQL

Delta Lake

Airflow

ADF

Git

CI/CD

Job description

Tata Consultancy Services in Jersey City is seeking a senior Data Engineer to design, develop, and maintain data pipelines using Databricks, Python, and PySpark. Collaborate with data architects, analysts, and business teams to deliver scalable data solutions for large volumes of structured and unstructured data.

You will optimize batch and streaming workflows, ensure data quality, governance, and security, and participate in code reviews and Agile ceremonies to drive continuous improvement.