Onsite_Data Engineer_ MA

Themesoft Inc; Quincy (MA); On-site; USD 140,000 - 170,000; Full time

Job summary

The Data Engineer - Databricks, Scala role at Themesoft Inc. in Quincy, MA requires 8+ years of data engineering experience with Databricks and Spark. Onsite, 6-12+ month engagement, focusing on data pipelines, streaming, and warehouse-style data processing.

Deep hands-on knowledge of Scala, PySpark/SQL, and Databricks components is essential for building robust data workflows and optimizing performance.

Qualifications

• 8+ years of data engineering with Databricks and Spark.
• 8+ years in hands-on Databricks solutions using scala and spark.
• Experience with data pipelines, SQL and processing frameworks.

Responsibilities

• Provide data ingestion, streaming and transformation solutions using Databricks.
• Develop and optimize Spark jobs and data transformations.

Skills

Databricks

Scala

Spark

SQL

Data pipelines

ETL/ELT

Data warehousing

Tools

Azure

AKS

Docker

Kubernetes

Job description

Job Title: Data Engineer - Databricks, Scala

Location: Quincy, MA- 100% ONSITE

Work Arrangement: Onsite

Duration: 6-12+ months

Databricks, Scala- Data Engineer

• Must have atleast 8+ years of extensive Data engineering experience in Databricks, ETL/ELT using data pipelines, SQL/Procesurr experience
• Must have 8 years of hands on development proficiency in implementing Databricks solutions using scala and spark with data frames and notebooks/sql
• Must have 8+ years of hands on development and performance tuning/ enhancement experience in scala/pyspark and sql
• Must have strong experience with Databricks, including developing and optimizing spark jobs, data transformations, and data processing workflows
• Must have experience and a good understanding on the below topics
• Databricks delta lake storage
• Unity catalog
• Autoloader
• Lakehouse architecture
• Medallion architecture
• Databricks serverless options
• Dataset and data frame
• Databricks clusters configuration and sizing
• Databricks job orchestration
• Spark structured streaming
• Must have experience with complex and large volume data streaming and data transformations using Azure Databricks
• Must have extensive experience in datawarehouse / data lakehouse implementation
• Must have experience with monitoring, debugging, and resolving issues in kive Databricks jobs
• Must have strong hands on expertise in troubleshooting devops pipelines, azure , and AKS services
• The candidate shd be able to provide data ingestion, streaming and transformations solutions and do development woek
• Experience with Apache big data implementation with programming proficiency like pyspark, R or Java prior to Databricks is advantageous
• Good to have experience with containerization technologies such as docker and Kubernetes

Work Authorization

• Applicants must be legally authorized to work in the United States.

Equal Employment Opportunity

• [Themesoft Inc] is an Equal Employment Opportunity employer. We consider qualified applicants without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, age, disability, veteran status, or any other status protected by applicable federal, state, or local law.

Skills :