Data Engineer

Compunnel, Inc.; Norfolk (VA); In loco; USD 110.000 - 150.000; Tempo pieno

Descrizione del lavoro

Compunnel, Inc. is seeking a Data Engineer to design, develop, optimize, and maintain scalable data platforms supporting enterprise analytics.

The role requires 5+ years of experience in data engineering, ETL, and data platform engineering, with strong SQL, Python, SSIS, Airflow, Spark, and Databricks skills. The engineer will collaborate with business stakeholders in an onsite environment to deliver reliable, secure data solutions, focusing on data quality, governance, and scalable

Competenze

• 5+ years of experience in Data Engineering, ETL Development, or Data Platform Engineering.
• Strong SQL skills with complex queries, data modeling, and performance tuning.
• Proficiency in Python for data engineering and ETL.
• Hands-on SSIS experience.
• Airflow workflow orchestration and Spark data processing expertise.
• Experience with Databricks and lakehouse architectures using Iceberg.
• Understanding of data warehousing, ETL/ELT concepts, and distributed data systems.
• Knowledge of data quality, monitoring, and troubleshooting techniques.
• Bachelor's degree in CS/IT/Engineering or related field.

Mansioni

• Design, develop, and optimize scalable data pipelines for batch and real-time processing.
• Build and maintain robust ETL/ELT solutions with Python, SQL, and SSIS.
• Develop and manage workflow orchestration using Apache Airflow.
• Implement data transformation and processing with Apache Spark.
• Design and support lakehouse architectures using Databricks and Apache Iceberg.
• Ensure data quality, security, and governance across platforms.
• Monitor, troubleshoot, and optimize ingestion and processing workflows.
• Collaborate with analysts, scientists, and architects to deliver scalable data solutions.
• Perform performance tuning of databases, Spark jobs, and pipelines.
• Support production deployments and maintenance for critical data systems.
• Follow coding, documentation, testing, and release-management best practices.

Conoscenze

Data engineering

Python

SQL

ETL/ELT

Apache Airflow

Apache Spark

Databricks

Apache Iceberg

SSIS

Data governance

Formazione

Bachelor's degree in Computer Science/IT/Engineering

Strumenti

Databricks

Apache Airflow

Apache Spark

Iceberg

SSIS

Descrizione del lavoro

The Data Engineer will design, develop, optimize, and maintain scalable data platforms and pipelines supporting enterprise analytics and business intelligence initiatives. The role requires strong expertise in modern data engineering technologies, cloud-based data platforms, ETL/ELT orchestration, distributed data processing, and lakehouse architectures. The engineer will work closely with business and technical stakeholders in an onsite environment to deliver reliable, secure, and scalable data solutions. The position requires 5+ years of experience in Data Engineering, ETL Development, or Data Platform Engineering.

Key Responsibilities

• Design, develop, and optimize scalable data pipelines for batch and real-time data processing.
• Build and maintain robust ETL/ELT solutions using Python, SQL, and SSIS.
• Develop and manage workflow orchestration using Apache Airflow.
• Implement data transformation and processing solutions using Apache Spark.
• Design, optimize, and support data lakehouse architectures using Databricks and Apache Iceberg.
• Ensure data quality, integrity, security, and governance across data platforms.
• Monitor, troubleshoot, and optimize data ingestion and processing workflows.
• Collaborate with business analysts, data scientists, architects, and application teams to understand data requirements and deliver scalable solutions.
• Perform performance tuning of databases, Spark jobs, and data pipelines.
• Support production deployments and provide ongoing maintenance for critical data systems.
• Follow best practices for coding, documentation, testing, and release management.

Required Qualifications

• 5+ years of experience in Data Engineering, ETL Development, or Data Platform Engineering.
• Strong experience with SQL, including complex query development, data modeling, and performance tuning.
• Proficiency in Python for data engineering and ETL development.
• Hands‑on experience with SSIS (SQL Server Integration Services).
• Experience designing and managing workflows using Apache Airflow.
• Strong knowledge of Apache Spark for large-scale data processing.
• Experience working with Apache Iceberg tables and lakehouse architectures.
• Hands‑on experience with Databricks for data engineering and analytics workloads.
• Strong understanding of data warehousing, ETL/ELT concepts, and distributed data systems.
• Experience with data quality, monitoring, and troubleshooting techniques.
• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.

Preferred Qualifications

• Experience with cloud platforms such as Azure, AWS, or GCP.
• Familiarity with CI/CD pipelines and DevOps practices for data platforms.
• Knowledge of data governance, metadata management, and security best practices.

Notes

Mandatory Areas

• Must Have Skills – Data Engineer
• Skill 1 – 8 + Years of exp in Data Engineering
• Skill 2 – 5 + Years of Exp in Python, ETL/ELT, SSIS
• Skill 3- 4+ Years of exp in AWS/ Azure /GCP
• Good To have Skills –
• Skill 1 – Yrs of Exp –N/A
• Skill 2 – Yrs of Exp –N/A
• Skill 3 – Yrs of Exp –N/A
• Skill 4 – Yrs of Exp -N/A
• Mandatory if Applicable
• Domain Experience (If any ) – N/A
• Must have Certifications –N/A
• Location – Norfolk, VA
• Onsite Requirement - Y/N- Y
• Number of days onsite – 3 Days