Hadoop Developer

Fixity Technologies; Charlotte (NC); On-site; USD 120,000 - 180,000; Full time

Job summary

Fixity Technologies seeks a highly experienced Hadoop Spark Developer to lead large-scale data processing initiatives. The role focuses on PySpark, Hadoop ecosystem, Hive, and Python, with a strong emphasis on building scalable ETL/ELT pipelines and optimizing Spark workloads.

The ideal candidate has 10+ years in distributed data processing, can collaborate with architects and data teams, and will leverage Copilot to accelerate development and testing. A bachelor's or master's degree is required.

Qualifications

• 10+ years of experience in Big Data technologies including Spark and Hadoop.
• Design, develop, optimize, and maintain large-scale data processing solutions.
• Bachelor's or master's degree in a related field.

Responsibilities

• Design, develop, and maintain scalable Big Data solutions using Hadoop and Spark.
• Build and optimize ETL/ELT pipelines with PySpark, Hive, and Python.
• Process and analyze large datasets in distributed environments.
• Develop high-performance Spark jobs and optimize workloads.
• Create and manage Hive tables, partitions, views, and complex queries.
• Implement data quality, validation, and reconciliation frameworks.
• Review code and enforce coding standards and best practices.
• Leverage Microsoft Copilot to accelerate development and testing.
• Build batch and real-time streaming apps with Kafka.
• Collaborate with Data Architects, Data Scientists, Analysts, and DevOps.
• Troubleshoot issues and perform root-cause analysis.
• Design data ingestion frameworks for structured, semi-structured, and unstructured data.
• Mentor junior developers and provide technical leadership.

Skills

PySpark

Hive

Python

SQL

Unix

Kafka

Java

Education

Bachelor's degree

Master's degree

Tools

Hadoop Ecosystem

Microsoft Copilot

Job description

Primary Skill: PySpark, Hive, Python, SQL

Secondary: Unix, Kafka, Java

We are seeking a highly experienced Hadoop Spark Developer with 10+ years of expertise in Big Data technologies, including PySpark, Hadoop Ecosystem, Hive, and Python. The ideal candidate will be responsible for designing, developing, optimizing, and maintaining large-scale data processing solutions.

Experience with Microsoft Copilot for AI-assisted development and productivity enhancement is highly desirable. The developer should hold a bachelor's or master's degree.

The candidate should possess strong analytical skills, hands-on experience in distributed data processing, and the ability to work closely with business stakeholders, architects, and data engineering teams.

• Design, develop, and maintain scalable Big Data solutions using Hadoop and Spark.
• Build and optimize ETL/ELT pipelines using PySpark, Hive, and Python.
• Process and analyze large datasets in distributed environments.
• Develop high-performance Spark jobs and optimize existing workloads.
• Create and manage Hive tables, partitions, views, and complex queries.
• Implement data quality, data validation, and reconciliation frameworks.
• Perform code reviews and ensure adherence to coding standards and best practices.
• Utilize Microsoft Copilot to accelerate development, automate code generation, troubleshooting, documentation, and testing activities.
• Strong experience building both batch and real-time streaming applications with Kafka.
• Collaborate with Data Architects, Data Scientists, Business Analysts, and DevOps teams.
• Troubleshoot production issues and perform root cause analysis.
• Design data ingestion frameworks for structured, semi-structured, and unstructured data.
• Mentor junior developers and provide technical leadership.