Data Engineer

VDart Inc; Dallas (TX); On-site; USD 95,000 - 130,000; Part time

Job summary

VDart Inc. is seeking a Data Engineer in Dallas, TX for a contract role. You will migrate data from on-premise DataLakes to AWS Lakehouse platforms, refactor pipelines, and ensure data integrity during large-scale transfers.

You will modernize SQL- and Spark-based processing for Snowflake and Iceberg, design validation frameworks, and collaborate with global teams to obtain migration sign-off.

Qualifications

• Minimum 3+ years of hands-on software development and/or data engineering experience.
• Experience with SQL development, analysis, query troubleshooting, and performance optimization.
• Programming in Python and/or Java for data processing or integration.
• Experience building or supporting ETL/ELT data pipelines and data migrations.
• Experience with distributed data processing using Apache Spark.
• Familiarity with data engineering concepts like SCD Type 2, schema evolution, partitioning, and data quality.

Responsibilities

• Perform end-to-end datastore migrations to AWS Lakehouse platforms as part of migration factory team.
• Refactor and migrate data pipelines including extraction logic, transformations, and scheduling.
• Execute large-scale data transfers ensuring data integrity and consistency across sources and targets.
• Modernize legacy SQL and Spark processing for Snowflake and Iceberg environments.
• Design data reconciliation and validation frameworks to verify data accuracy.
• Collaborate with stakeholders and teams to validate migration and obtain sign-off.

Skills

SQL

Python/Java

ETL/ELT pipelines

Apache Spark

Snowflake

Apache Iceberg

Kafka

CI/CD

Kubernetes

Tools

Snowflake

Apache Iceberg

Apache Spark

Kafka

Hadoop

Sybase IQ

Job description

Role: Data Engineer

Location: Dallas, TX (Onsite)

Type: Contract

Day to Day Job Duties:

• Perform end-to-end datastore migrations from on-premises DataLake environments to AWS-hosted Lakehouse platforms as part of the migration factory team.
• Refactor and migrate existing data pipelines, including data extraction logic, transformation processes, and job scheduling.
• Execute large-scale data transfers while ensuring data integrity, completeness, reliability, and consistency between source and target platforms.
• Translate and modernize legacy SQL and Apache Spark-based processing logic for Snowflake and Apache Iceberg environments.
• Analyze existing data usage patterns, business requirements, and downstream consumption to support the development and delivery of reusable data products.
• Design and build data reconciliation and validation frameworks to verify data accuracy during and after migration.
• Collaborate with business stakeholders, application teams, data owners, and technical teams to perform validation and obtain migration sign-off.
• Act as a technical liaison between migration, data engineering, application, infrastructure, and business teams throughout the migration lifecycle.
• Troubleshoot data pipeline, data transformation, performance, and migration-related issues and implement appropriate technical solutions.
• Optimize data pipelines and processing workloads to improve performance, scalability, and operational efficiency.
• Apply core data engineering concepts including Slowly Changing Dimensions (SCD Type 2), schema evolution, partitioning, clustering, normalization and denormalization, natural and surrogate keys, and data quality controls.
• Work with structured and semi-structured data formats including JSON, Avro, and Parquet.
• Follow established Software Development Life Cycle (SDLC), source-control, testing, deployment, and Continuous Integration/Continuous Deployment (CI/CD) practices.
• Adapt to new data technologies, migration tools, engineering standards, and workflows as required by the program.
• Collaborate effectively with geographically distributed and global delivery teams.

Basic Qualifications:

• Minimum 3+ years of hands-on software development and/or data engineering experience, including coding, development, troubleshooting, and implementation of data-related solutions.
• Minimum 3+ years of experience working with SQL, including development, analysis, query troubleshooting, and performance optimization.
• Minimum 3+ years of hands‑on programming experience using Python and/or Java for data processing, application development, automation, or integration.
• Minimum 3+ years of experience developing or supporting ETL/ELT data pipelines, including data extraction, transformation, loading, and pipeline troubleshooting.
• Experience developing or supporting distributed data-processing solutions using Apache Spark.
• Experience working with data engineering concepts including SCD Type 2, schema evolution, partitioning, clustering, normalization versus denormalization, natural versus surrogate keys, and data quality frameworks.
• Experience working with one or more data and integration technologies including Kafka, ANSI SQL, FTP, Apache Spark, Hadoop, Snowflake, Apache Iceberg, and Sybase IQ.
• Experience working with structured and semi-structured data formats including JSON, Avro, and Parquet.
• Experience working within established SDLC and CI/CD processes, including source control, testing, deployment, and release practices.
• Familiarity with containerized application environments and Kubernetes.
• Demonstrated ability to troubleshoot technical issues, communicate effectively with stakeholders, collaborate across global teams, and take ownership of assigned deliverables.

Nice to Have:

• Experience performing large-scale data platform or datastore migration initiatives.
• Experience migrating data workloads from on-premises environments to cloud-based platforms, particularly AWS.
• Experience working with Lakehouse architectures.
• Experience with Snowflake and Apache Iceberg-based data platforms.
• Experience working within the financial services industry.
• Experience supporting complex migration programs involving multiple business, technology, and global delivery teams.