Big Data Developer

eNcloud; Katy (TX); Hybrid; USD 95,000 - 140,000; Full time

Job summary

eNcloud Services LLC in Katy, TX seeks Big Data Developer to build data transformation and ingestion pipelines using Spark, Hive, HDFS, and related tools. You will implement ETL processes, write complex SQL, and ensure data quality across large datasets in analytics workloads.

The role requires a Bachelor’s degree and 5+ years of experience, with travel to client locations possible. Strong scripting and CI/CD experience are a plus.

Qualifications

• Bachelor’s degree in Computer Science, IT, Engg or related field.
• At least 60 months of experience in Big Data / data engineering environments.
• Travel and relocation to client locations may be required.

Responsibilities

• Build data transformation, metadata, and workload management processes.
• Ingest and curate data using Spark, Hive, HDFS, Sqoop, and related tools.
• Develop high-performance ETL code with strong SQL and data quality checks.
• Write shell scripts; run complex SQL queries and Hadoop commands.
• Monitor production jobs and support CI/CD and version control.

Skills

Spark

Hive

HDFS

Sqoop

HBase

Kerberos

Sentry

Impala

ETL

SQL

Python

Scala

Education

Bachelor's degree in Computer Science / IT / Engg

Tools

Bitbucket

CI/CD

Job description

eNcloud Services LLC, a Katy, TX based IT solutions firm. Multiple openings for Big Data Developer.

Job Duties and Responsibilities:
Build processes that support data transformation, data structures, metadata, dependency and workload management
Build and Implement data ingestion and curation process developed using Big data tools such as Spark (Scala/python), Hive, Spark, HDFS, Sqoop, Hbase, Kerberos, Sentry and Impala etc.
Handle ingestion of huge volumes data from various platforms for Analytics needs and writing high-performance, reliable and maintainable ETL code Strong SQL knowledge and data analysis skills for data anomaly detection and data quality assurance.
Write shell scripts. Complex SQL queries, Hadoop commands and Git.
Create Database, Schemas, Hive tables (External and Managed) with various file formats (Orc, Parquet, Avro and Text etc.)
Monitor performance of production jobs and advising any necessary infrastructure changes.
Work on Code version using Bitbucket and CI/CD pipeline.
Requirements: Bachelor’s degree in Computer Science, IT, Engg or related with at least 60 months of experience. Travel and relocation to various unanticipated client locations throughout the United States may be required.Equal Opportunity Employer.

Job Category

IT