Data engineer

MACHINE LEARNING TECHNOLOGIES LLC; Quincy (MA); On-site; USD 83,000 - 110,000; Full time

Job summary

MACHINE LEARNING TECHNOLOGIES LLC in Quincy, MA is seeking a Data Engineer to build and optimize Spark-based data pipelines and Azure Databricks solutions. You will design data lakehouse architectures, implement streaming, and tune SQL performance within large-scale environments.

Responsibilities include troubleshooting live Databricks jobs and DevOps pipelines while leveraging Docker/Kubernetes where applicable. Collaboration across IT/Software domains is expected.

Qualifications

• Develop and optimize Spark jobs and data pipelines.
• Azure Databricks for large-volume processing and streaming.
• Experience with data warehouse/lakehouse design and Delta Lake.
• Debug live Databricks jobs and monitor pipelines.
• Knowledge of Docker and Kubernetes is advantageous.

Responsibilities

• Develop and optimize Spark jobs, data transformations, and processing workflows.
• Implement streaming and transformation solutions using Azure Databricks.
• Design and implement data warehouse/lakehouse solutions.
• Monitor, debug, and resolve issues in live Databricks jobs.
• Troubleshoot DevOps pipelines, Azure, and AKS services.
• Build data ingestion, streaming, and transformation solutions.

Skills

Apache Spark

Spark DataFrames

Notebooks

Spark SQL

ETL/ELT

Scala

PySpark

SQL tuning

Azure Databricks

Delta Lake

Unity Catalog

Auto Loader

Lakehouse

Medallion Architecture

Databricks Serverless

Dataset

Cluster configuration

Databricks Job Orchestration

Spark Structured Streaming

Docker

Kubernetes

Tools

Docker

Kubernetes

Job description

Job Details

• Job ID: J55183
• Job Title: Data engineer
• Location: Quincy, MA
• Duration: 10 Months + Extension
• Duration: 14 Months
• Domain Exposure: IT/Software
• Client: To Be Discussed Later
• Employment Type: W-2, 1099, C2C
• Bill Rate: $68/hour (C2C)
• Job Type: W-2, 1099, C2C
• Work Authorization: US Citizen, Green Card, H-1B, OPT-EAD, CPT, Other
• Work Authorization: US Citizen, Green Card, OPT-EAD, CPT, H-1B, H4-EAD, Other

Mandatory Technical Skills

• Apache Spark, Spark DataFrames, Notebooks and Spark SQL
• ETL/ELT and data pipeline development
• Scala/PySpark and SQL performance tuning
• Azure Databricks and large-volume data processing
• Databricks Expertise
• Delta Lake Storage
• Unity Catalog
• Auto Loader
• Lakehouse and Medallion Architecture
• Databricks Serverless Options
• Dataset and DataFrame
• Cluster Configuration and Sizing
• Databricks Job Orchestration
• Spark Structured Streaming

Additional Responsibilities & Requirements

• Develop and optimize Spark jobs, data transformations, and processing workflows.
• Implement complex data streaming and transformation solutions using Azure Databricks.
• Design and implement data warehouse/lakehouse solutions.
• Monitor, debug, and resolve issues in live Databricks jobs.
• Troubleshoot DevOps pipelines, Azure, and AKS services.
• Build data ingestion, streaming, and transformation solutions.
• Experience with Apache Big Data technologies and languages such as PySpark, R, or Java is advantageous.
• Docker and Kubernetes experience is good to have.

Equal Opportunity Employer

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.