Data engineer

Valiant Technologies LLC; Quincy (MA); On-site; USD 77,000 - 110,000; Full time

Job summary

Valiant Technologies LLC is seeking a Data Engineer in Quincy, MA, to develop and optimize Spark-based data processing and streaming pipelines on Azure Databricks. The role involves design and implementation of lakehouse architectures and ongoing production support for Databricks jobs.

Ideal candidates have strong PySpark/Scala skills, SQL tuning experience, and familiarity with DevOps and cloud services. Experience with Docker/Kubernetes is a plus for deployment and scalability.

Qualifications

• Experience with Spark-based data processing and ETL pipelines.
• Implement streaming data solutions using Databricks.
• Design data warehouse/lakehouse architectures.
• Monitor and debug Databricks jobs in production.
• Familiarity with DevOps pipelines and cloud services.

Responsibilities

• Develop and optimize Spark jobs, data transformations, and processing workflows.
• Implement complex data streaming and transformation solutions using Azure Databricks.
• Design and implement data warehouse/lakehouse solutions.
• Monitor, debug, and resolve issues in live Databricks jobs.
• Troubleshoot DevOps pipelines, Azure, and AKS services.
• Build data ingestion, streaming, and transformation solutions.

Skills

Spark DataFrames

ETL/ELT pipelines

Scala/PySpark

SQL performance tuning

Streaming data

Tools

Azure Databricks

Databricks Unity Catalog

Databricks Lakehouse

Databricks Job Orchestration

Docker

Kubernetes

Job description

US Citizen, Green Card, H-1B, OPT-EAD, CPT, Other

Job Title: Data engineer

Location: Quincy, MA

Duration: 14 Months

Bill Rate: $68/hour (C2C)

Job Type: W-2, 1099, C2C

Client: To Be Discussed Later

Work Authorization: US Citizen, Green Card, OPT-EAD, CPT, H-1B, H4-EAD, Other

Job Description

Mandatory Technical Skills

• Apache Spark, Spark DataFrames, Notebooks and Spark SQL
• ETL/ELT and data pipeline development
• Scala/PySpark and SQL performance tuning
• Azure Databricks and large-volume data processing
• Databricks Expertise
• Unity Catalog
• Auto Loader
• Lakehouse and Medallion Architecture
• Databricks Serverless Options
• Dataset and DataFrame
• Cluster Configuration and Sizing
• Databricks Job Orchestration
• Spark Structured Streaming

Additional Responsibilities & Requirements

• Develop and optimize Spark jobs, data transformations, and processing workflows.
• Implement complex data streaming and transformation solutions using Azure Databricks.
• Design and implement data warehouse/lakehouse solutions.
• Monitor, debug, and resolve issues in live Databricks jobs.
• Troubleshoot DevOps pipelines, Azure, and AKS services.
• Build data ingestion, streaming, and transformation solutions.
• Experience with Apache Big Data technologies and languages such as PySpark, R, or Java is advantageous.
• Docker and Kubernetes experience is good to have.

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.