Senior Data Engineer

Gemburg Inc.; Charlotte (NC); Hybrid; USD 96,000 - 165,000; Part time

Job summary

Gemburg Inc. in Charlotte, North Carolina, seeks an experienced Senior Data Engineer to design and maintain scalable data pipelines using PySpark, Databricks, Python and SQL. The role focuses on ETL/ELT development and large-volume data processing with performance optimization.

The ideal candidate will write clean, production-ready code, collaborate with data architects and analysts, and contribute to reliable data solutions in a hybrid contract setting.

Qualifications

• 10+ years of experience in data engineering or related software/data development roles.
• Strong hands-on experience with Python and PySpark.
• Strong SQL skills, including complex queries, joins, aggregations, and query optimization.
• Experience building and maintaining ETL/ELT data pipelines.
• Knowledge of Apache Spark architecture, distributed processing, and performance tuning.
• Experience with Delta Lake and lakehouse architecture.
• Strong coding, debugging, troubleshooting, and problem-solving skills.
• Experience with Git and software development best practices.
• Familiarity with cloud platforms such as AWS, Azure, or GCP.
• Strong communication and collaboration skills.

Responsibilities

• Design, develop, and maintain scalable data pipelines using PySpark and Databricks.
• Write, test, debug, and optimize Python, PySpark, and SQL code.
• Build and maintain ETL/ELT processes for structured and unstructured data.
• Develop data transformations and workflows to process large volumes of data.
• Work with Databricks notebooks, Delta Lake, and Spark-based data processing frameworks.
• Optimize Spark jobs, troubleshoot performance issues, and improve pipeline efficiency.
• Implement data quality checks, validation, error handling, and monitoring.
• Integrate data from multiple sources, including databases, APIs, cloud storage, and enterprise systems.
• Collaborate with data architects, analysts, and business stakeholders to deliver reliable data solutions.
• Follow coding standards, version control practices, testing procedures, and CI/CD processes.

Skills

Python

PySpark

SQL

ETL/ELT

Delta Lake

Apache Spark

Databricks

Git

Cloud platforms (AWS/Azure/GCP)

CI/CD

Agile

Tools

Databricks

Airflow

Azure Data Factory

Databricks Workflows

Job description

Location: Charlotte, North Carolina (Hybrid)// Need only locals

Experience: 10+ Years

Job Type: Contract

Job Summary

We are seeking an experienced Senior Data Engineer with strong hands-on expertise in PySpark, Databricks, Python, and SQL to design, develop, and maintain scalable data engineering solutions. The ideal candidate will have extensive experience building data pipelines, processing large datasets, developing ETL/ELT workflows, and optimizing data processing performance.

The candidate must possess strong programming and problem-solving skills, with the ability to write clean, efficient, and production-ready code.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using PySpark and Databricks.
• Write, test, debug, and optimize Python, PySpark, and SQL code.
• Build and maintain ETL/ELT processes for structured and unstructured data.
• Develop data transformations and workflows to process large volumes of data.
• Work with Databricks notebooks, Delta Lake, and Spark-based data processing frameworks.
• Optimize Spark jobs, troubleshoot performance issues, and improve pipeline efficiency.
• Implement data quality checks, validation, error handling, and monitoring.
• Integrate data from multiple sources, including databases, APIs, cloud storage, and enterprise systems.
• Collaborate with data architects, analysts, and business stakeholders to deliver reliable data solutions.
• Follow coding standards, version control practices, testing procedures, and CI/CD processes.

Required Skills

• 10+ years of experience in data engineering or related software/data development roles.
• Strong hands-on experience with Python and PySpark.
• Strong SQL skills, including complex queries, joins, aggregations, and query optimization.
• Experience building and maintaining ETL/ELT data pipelines.
• Knowledge of Apache Spark architecture, distributed processing, and performance tuning.
• Experience with Delta Lake and lakehouse architecture.
• Strong coding, debugging, troubleshooting, and problem-solving skills.
• Experience with Git and software development best practices.
• Familiarity with cloud platforms such as AWS, Azure, or GCP.
• Strong communication and collaboration skills.

Preferred Qualifications

• Experience with Azure Databricks, AWS Databricks, or another cloud-based data platform.
• Knowledge of data warehousing, dimensional modeling, and data lake architecture.
• Experience with orchestration tools such as Airflow, Azure Data Factory, or Databricks Workflows.
• Familiarity with CI/CD, automated testing, and deployment practices.
• Experience working in Agile development environments.