Big Data Developer

eNcloud; Boston, Northern (MA, KY); Hybrid; USD 120,000 - 180,000; Full time

Job summary

eNcloud is seeking a data engineer in Boston to build data transformation and ingestion pipelines. You will develop ETL processes using Spark, Hive, HDFS and related tools, while handling large data volumes and ensuring data quality.

You will write high-performance SQL, shell scripts, and manage Hive tables and data structures. Collaboration with CI/CD and Bitbucket is required, with travel to client locations as needed.

Qualifications

• Bachelor’s degree in Computer Science, IT, Engg or related
• 5+ years of experience in data engineering or related role
• Experience with big data tools and data ingestion/curation processes

Responsibilities

• Build processes for data transformation, data structures, metadata, dependency and workload management
• Develop data ingestion and curation using Spark, Hive, HDFS, Sqoop, HBase, Kerberos and Impala
• Ingest large data volumes from multiple platforms and write high-performance ETL code with strong SQL and data quality focus
• Write shell scripts, complex SQL queries, Hadoop commands and Git workflows
• Create databases, schemas, Hive tables (External and Managed) with Orc/Parquet/Avro/Text formats
• Monitor production job performance and advise infrastructure changes
• Work with Bitbucket and CI/CD pipelines

Skills

Big data

Spark

Hive

SQL

ETL

Shell scripting

Git

CI/CD

Bitbucket

Hadoop

Education

Bachelor's degree

Tools

HDFS

Sqoop

HBase

Kerberos

Sentry

Impala

Job description

Job Duties and Responsibilities

• Build processes that support data transformation, data structures, metadata, dependency and workload management
• Build and Implement data ingestion and curation process developed using Big data tools such as Spark (Scala/python), Hive, Spark, HDFS, Sqoop, Hbase, Kerberos, Sentry and Impala etc.
• Handle ingestion of huge volumes data from various platforms for Analytics needs and writing high-performance, reliable and maintainable ETL code Strong SQL knowledge and data analysis skills for data anomaly detection and data quality assurance.
• Write shell scripts. Complex SQL queries, Hadoop commands and Git.
• Create Database, Schemas, Hive tables (External and Managed) with various file formats (Orc, Parquet, Avro and Text etc.)
• Monitor performance of production jobs and advising any necessary infrastructure changes.
• Work on Code version using Bitbucket and CI/CD pipeline.
• Requirements: Bachelor’s degree in Computer Science, IT, Engg or related with at least 60 months of experience. Travel and relocation to various unanticipated client locations throughout the United States may be required. Equal Opportunity Employer.

Job Category

IT