Data Engineer

Tata Consultancy Services Limited; Quincy (MA); On-site; USD 110,000 - 126,000; Full time

Job summary

Tata Consultancy Services Limited in Quincy, MA is seeking an experienced Engineer to lead data engineering initiatives using Databricks and Spark. The role focuses on building scalable data pipelines, streaming solutions, and lakehouse architectures to support analytics and decision making.

Candidates should have 8+ years in data engineering with Scala, PySpark, SQL, and hands-on Databricks implementation. Familiarity with DevOps, Azure services, and containerization is a plus.

Qualifications

• 8+ years of data engineering experience with Databricks and ETL/ELT data pipelines.
• SQL experience and processing pipelines.
• Hands-on Databricks with Scala and Spark (dataframes, notebooks, SQL).
• 8+ years in Scala/PySpark/SQL development and performance tuning.
• Databricks delta lake storage, Unity catalog, Autoloader and lakehouse architectures.
• Experience with Databricks serverless options, clusters sizing and job orchestration.
• Familiarity with Spark structured streaming.

Responsibilities

• Develop complex data streaming and transformations using Azure Databricks.
• Implement data warehouse/data lakehouse solutions.
• Monitor, debug and resolve issues in Databricks jobs.
• Troubleshoot DevOps pipelines, Azure, and AKS services.
• Provide ingestion, streaming and transformation solutions and development work.
• Work with PySpark, R, or Java in big data environments.
• Familiarity with containerization (Docker, Kubernetes) is a plus.

Skills

Scala

SQL

Education

Bachelor of Computer Science

Job description

Must Have Technical/Functional Skills

• Must have atleast 8+ years of extensive Data engineering experience in Databricks, ETL/ELT using data pipelines.
• SQL/Procesurr experience.
• Must have 8 years of hands-on development proficiency in implementing Databricks solutions using scala and spark with data frames and notebooks/sql
• Must have 8+ years of hands-on development and performance tuning/ enhancement experience in scala/pyspark and sql
• Must have strong experience with Databricks, including developing and optimizing spark jobs, data transformations, and data processing workflows
• Must have experience and a good understanding on the above topics
• Databricks delta lake storage
• Unity catalog
• Autoloader
• Lakehouse architecture
• Medallion architecture
• Databricks serverless options
• Dataset and dataframe
• Databricks clusters configuration and sizing
• Databricks job orchestration
• Spark structured streaming

Roles & Responsibilities

• Must have experience with complex and large volume data streaming and data transformations using Azure Databricks
• Must have extensive experience in datawarehouse / data lakehouse implementation
• Must have experience with monitoring, debugging, and resolving issues in kive Databricks jobs
• Must have strong hands on expertise in troubleshooting devops pipelines, azure , and AKS services
• The candidate shd be able to provide data ingestion, streaming and transformations solutions and do development work
• Experience with apache big data implementation with programming proficiency like pyspark, R or Java prior to
• Databricks is advantageous
• Good to have experience with containerization technologies such as docker and kubernetes

TCS Employee Benefits Summary:

• Discretionary Annual Incentive.
• Comprehensive Medical Coverage: Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans.
• Family Support: Maternal & Parental Leaves.
• Insurance Options: Auto & Home Insurance, Identity Theft Protection.
• Convenience & Professional Growth: Commuter Benefits & Certification & Training Reimbursement.
• Time Off: Vacation, Time Off, Sick Leave & Holidays.
• Legal & Financial Assistance: Legal Assistance, 401K Plan, Performance Bonus, College Fund, Student Loan Refinancing.
• Salary Range: $110,000-$126,000 a year

Location

Quincy, MA

Function

TECHNOLOGY

Role

Engineer

ID

435520

Skills

Scala | SQL

Salary Range

$110,000 - $126,000

Desired Candidate Profile

Qualifications

BACHELOR OF COMPUTER SCIENCE