Big Data Developer Sr

eNcloud; Hyderabad; On-site; INR 1,800,000 - 2,600,000; Full time

Job summary

eNcloud in Hyderabad, India seeks an experienced Data Engineer specializing in ETL and big data. You will design and implement data ingestion pipelines, optimize data transformation, and ensure data quality for analytics projects.

The role requires at least 5 years of experience, a Bachelors in CS/IT/Engineering, and readiness to travel to client locations in the United States. You will work with Spark, Hive, HDFS, and related technologies in a fast-paced environment.

Qualifications

• Bachelor’s degree in Computer Science, IT, Engg or related.
• 5+ years of experience in data engineering or ETL roles.
• Willingness to travel and relocate to client locations in the United States as required.
• Equal Opportunity Employer.

Responsibilities

• Build processes to support data transformation, data structures, metadata, and workload management.
• Develop data ingestion and curation using Spark, Hive, HDFS and related tools.
• Handle high-volume data ingestion for analytics and write high-performance ETL code with strong SQL.
• Write shell scripts, complex SQL, Hadoop commands and use Git in CI/CD pipelines.
• Create Hive tables and schemas with Orc/Parquet/Avro/text formats and monitor production job performance.

Skills

ETL development

SQL

Data transformation

Data quality

Education

Bachelor’s degree in CS/IT/Engineering

Tools

Spark

Hive

HDFS

Sqoop

HBase

Kerberos

Sentry

Impala

Git

CI/CD

Job description

Job Duties and Responsibilities

• Build processes that support data transformation, data structures, metadata, dependency and workload management
• Build and Implement data ingestion and curation process developed using Big data tools such as Spark (Scala/python), Hive, Spark, HDFS, Sqoop, Hbase, Kerberos, Sentry and Impala etc.
• Handle ingestion of huge volumes data from various platforms for Analytics needs and writing high-performance, reliable and maintainable ETL code Strong SQL knowledge and data analysis skills for data anomaly detection and data quality assurance.
• Write shell scripts. Complex SQL queries, Hadoop commands and Git.
• Create Database, Schemas, Hive tables (External and Managed) with various file formats (Orc, Parquet, Avro and Text etc.)
• Monitor performance of production jobs and advising any necessary infrastructure changes.
• Work on Code version using Bitbucket and CI/CD pipeline.
• Requirements: Bachelor’s degree in Computer Science, IT, Engg or related with at least 60 months of experience. Travel and relocation to various unanticipated client locations throughout the United States may be required. Equal Opportunity Employer.

Job Category

IT