Data Engineer ETL

Compunnel, Inc.; Durham (NC); On-site; USD 100,000 - 130,000; Full time

Job summary

Compunnel, Inc. is seeking a highly motivated Data Engineer (ETL) to design, develop, and maintain scalable data engineering solutions. The role includes building high-performance ETL pipelines, developing applications using Apache Spark, and processing large-scale datasets with cloud-native technologies.

Ideal candidates have strong expertise in Spark, Scala, and Java, while collaborating with cross-functional teams. The company values innovation and provides an environment for continuous improvement in data engineering practices.

Qualifications

• 10+ years of experience developing Spark services for large-scale data movement.
• Experience scheduling, monitoring, debugging, and optimizing ETL Spark batch processes.
• Hands-on experience developing Java clients for REST and SOAP API integrations.

Responsibilities

• Design, develop, and maintain scalable ETL pipelines for enterprise data integration.
• Develop, test, deploy, and support ETL batch jobs using Apache Spark.
• Collaborate with cross-functional teams to design innovative data solutions.

Skills

Apache Spark

Scala

Java

ETL processes

Cloud platforms

Big data technologies

Agile methodologies

Education

Bachelor's degree in Computer Science or related field

Tools

Apache Spark

Amazon EMR

NoSQL databases (Cassandra/Yugabyte)

Snowflake

REST/SOAP APIs

Parquet/HDF5

Job description

We are seeking a highly motivated Data Engineer (ETL) to design, develop, and maintain scalable data engineering solutions supporting enterprise data aggregation and integration initiatives. This role focuses on building high-performance ETL pipelines, developing Apache Spark applications, processing large-scale datasets, and leveraging cloud-native technologies to deliver reliable and efficient data solutions. The ideal candidate will have strong expertise in Spark, Scala, Java, cloud platforms, and big data technologies while collaborating with cross-functional teams in an Agile environment.

Key Responsibilities

• Design, develop, and maintain scalable ETL pipelines for enterprise data integration and movement.
• Develop, test, deploy, and support ETL batch jobs using Apache Spark and Amazon EMR.
• Build and maintain Spark services for large-scale data processing.
• Schedule, monitor, troubleshoot, and optimize Spark batch processing jobs.
• Develop Java clients for consuming REST and SOAP APIs.
• Develop and maintain Scala-based Spark batch applications.
• Process and transform large datasets using Apache Spark.
• Store and manage processed data in NoSQL databases such as Cassandra or Yugabyte.
• Design and implement cloud-based data warehouse and data lake solutions using Snowflake.
• Work with large-scale data storage formats including Parquet and HDF5.
• Collaborate with cross-functional teams to design innovative data solutions that address business requirements.
• Participate in code reviews, pair programming, and technical discussions to ensure code quality and best practices.
• Support the full Software Development Life Cycle (SDLC), including analysis, design, development, testing, deployment, and maintenance.
• Troubleshoot production issues and optimize data processing performance.
• Identify opportunities for innovation and continuous improvement in data engineering practices.
• Collaborate across multiple Agile teams to deliver scalable and reliable data solutions.
• Apply a customer-first mindset when designing and delivering enterprise data solutions.

Required Qualifications

• Bachelor's degree in Computer Science, Information Systems, or a related field.
• 10+ years of experience developing Spark services for large-scale data movement.
• Experience scheduling, monitoring, debugging, and optimizing ETL Spark batch processes.
• Hands-on experience developing Java clients for REST and SOAP API integrations.
• Experience developing Scala Spark batch applications.
• Experience developing, deploying, and maintaining Apache Spark and Amazon EMR ETL jobs.
• Experience processing big data using Apache Spark.
• Experience with NoSQL databases such as Cassandra or Yugabyte.
• Experience with cloud-based data warehouse and data lake platforms such as Snowflake.
• Experience managing large datasets using storage formats such as Parquet and HDF5.
• Strong understanding of software development lifecycle (SDLC) methodologies.
• Strong analytical, troubleshooting, and problem-solving skills.
• Experience working in Agile software development environments.
• Excellent collaboration and communication skills.

Preferred Qualifications

• Experience designing enterprise-scale data integration and aggregation solutions.
• Experience supporting external-facing data platforms and APIs.
• Experience working with financial services or FinTech data platforms.
• Passion for continuous learning and adopting modern data engineering best practices.