Big Data Engineer: Spark, ETL & Data Quality

eNcloud Services LLC; Katy (TX); On-site; USD 100,000 - 170,000; Full time

Job summary

eNcloud Services LLC is seeking a data engineer to build scalable ETL pipelines and big data processing solutions. You will design data ingestion, transformation and curation processes using Spark, Hive, HDFS, and related tools.

The role requires strong SQL, data analysis skills, and experience with Git and CI/CD workflows. The position involves handling large data volumes across client locations in the United States, with travel as needed and a commitment to data quality and reliable pipeline

Qualifications

• Bachelor’s degree in Computer Science, IT, Engineering or related field with at least 60 months of experience.
• Travel and relocation to various client locations throughout the United States may be required.

Responsibilities

• Build processes that support data transformation, data structures, metadata, dependency and workload management.
• Build and implement data ingestion and curation processes developed using Big data tools such as Spark (Scala/Python), Hive, HDFS, Sqoop, HBase, Kerberos, Sentry and Impala.
• Handle ingestion of large volumes of data from various platforms for analytics needs and write high-performance, reliable and maintainable ETL code.
• Demonstrate strong SQL knowledge and data analysis skills for data anomaly detection and data quality assurance.
• Write shell scripts along with complex SQL queries, Hadoop commands and use Git.
• Create database schemas and Hive tables (external and managed) with various file formats (Orc, Parquet, Avro, Text, etc.).
• Monitor performance of production jobs and advise on necessary infrastructure changes.
• Work on code versioning using Bitbucket and CI/CD pipeline.

Skills

Data transformation

Data ingestion

Data quality assurance

SQL knowledge

Data analysis

CI/CD

Education

Bachelor's degree in Computer Science/IT/Engineering or related field

Tools

Spark

Hive

HDFS

Sqoop

HBase

Kerberos

Sentry

Impala

Bitbucket

Git

CI/CD pipeline

Job description

eNcloud Services LLC is seeking a data engineer to build scalable ETL pipelines and big data processing solutions. You will design data ingestion, transformation and curation processes using Spark, Hive, HDFS, and related tools.

The role requires strong SQL, data analysis skills, and experience with Git and CI/CD workflows. The position involves handling large data volumes across client locations in the United States, with travel as needed and a commitment to data quality and reliable pipeline