Big Data Engineer: Spark, ETL, Data Quality

eNcloud; Boston, Northern (MA, KY); Hybrid; USD 120,000 - 180,000; Full time

Job summary

eNcloud is seeking a data engineer in Boston to build data transformation and ingestion pipelines. You will develop ETL processes using Spark, Hive, HDFS and related tools, while handling large data volumes and ensuring data quality.

You will write high-performance SQL, shell scripts, and manage Hive tables and data structures. Collaboration with CI/CD and Bitbucket is required, with travel to client locations as needed.

Qualifications

• Bachelor’s degree in Computer Science, IT, Engg or related
• 5+ years of experience in data engineering or related role
• Experience with big data tools and data ingestion/curation processes

Responsibilities

• Build processes for data transformation, data structures, metadata, dependency and workload management
• Develop data ingestion and curation using Spark, Hive, HDFS, Sqoop, HBase, Kerberos and Impala
• Ingest large data volumes from multiple platforms and write high-performance ETL code with strong SQL and data quality focus
• Write shell scripts, complex SQL queries, Hadoop commands and Git workflows
• Create databases, schemas, Hive tables (External and Managed) with Orc/Parquet/Avro/Text formats
• Monitor production job performance and advise infrastructure changes
• Work with Bitbucket and CI/CD pipelines

Skills

Big data

Spark

Hive

SQL

ETL

Shell scripting

Git

CI/CD

Bitbucket

Hadoop

Education

Bachelor's degree

Tools

HDFS

Sqoop

HBase

Kerberos

Sentry

Impala

Job description

eNcloud is seeking a data engineer in Boston to build data transformation and ingestion pipelines. You will develop ETL processes using Spark, Hive, HDFS and related tools, while handling large data volumes and ensuring data quality.

You will write high-performance SQL, shell scripts, and manage Hive tables and data structures. Collaboration with CI/CD and Bitbucket is required, with travel to client locations as needed.