Senior Data Engineer

Yorkshire Global Solutions Inc.; United States; Remote; USD 130,000 - 180,000; Full time

Job summary

Yorkshire Global Solutions Inc. is seeking a Senior Data Engineer to design, develop, and maintain scalable data pipelines, data platforms, and cloud-based data solutions.

The role emphasizes strong hands-on experience with Python, SQL, ETL/ELT, Apache Spark, PySpark, Databricks, Snowflake, and cloud technologies. You will build reliable batch and real-time pipelines, develop data transformations, integrate data from multiple sources, optimize workloads, and support enterprise data platforms

Qualifications

• 10+ years of experience in Data Engineering, Data Development, or related data-focused roles.

Responsibilities

• Design, develop, and maintain scalable data pipelines and data platforms.

Skills

Python

SQL

ETL/ELT

Apache Spark

PySpark

Databricks

Snowflake

Azure/AWS

Data Pipelines

Data Warehousing

Kafka

Airflow

Delta Lake

Git

CI/CD

Tools

Python

SQL

ETL/ELT

Apache Spark

PySpark

Databricks

Snowflake

Azure

AWS

Kafka

Airflow

Delta Lake

Git

CI/CD

Job description

Job Title: Senior Data Engineer (W2 Only)

Location: Remote

Experience Level: 10+ Years

Type: Contract - W2

Job Summary

We are looking for an experienced Senior Data Engineer to design, develop, and maintain scalable data pipelines, data platforms, and cloud-based data solutions. The ideal candidate will have strong hands-on experience with Python, SQL, ETL/ELT, Apache Spark, PySpark, Databricks, Snowflake, and cloud technologies.

The candidate will be responsible for building reliable batch and real-time data pipelines, developing data transformations, integrating data from multiple sources, optimizing data processing workloads, and supporting enterprise data platforms. The ideal candidate should have strong experience with modern data engineering architectures, cloud data services, data warehousing, distributed processing, and CI/CD practices.

Must-Have Technologies / Tools

Python | SQL | ETL/ELT | Apache Spark | PySpark | Databricks | Snowflake | Azure/AWS | Data Pipelines | Data Warehousing | Kafka | Airflow | Delta Lake | Git | CI/CD

Required Skills

• 10+ years of experience in Data Engineering, Data Development, or related data-focused roles.
• Strong hands-on experience designing, developing, and maintaining enterprise-scale data pipelines.
• Strong programming experience using Python for data engineering and data processing.
• Strong SQL development experience including complex queries, joins, CTEs, window functions, stored procedures, and performance optimization.
• Strong experience with ETL/ELT development and data integration processes.
• Extensive experience working with Apache Spark and PySpark for large-scale distributed data processing.
• Strong experience with Databricks and modern cloud-based data platforms.
• Experience developing scalable data processing solutions using Delta Lake or similar technologies.
• Strong experience with Snowflake or other modern cloud data warehouse platforms.
• Experience designing and implementing dimensional data models, including fact and dimension tables.
• Strong understanding of data warehousing concepts, data lakes, data lakehouses, and modern data architectures.
• Experience developing batch and near-real-time data pipelines.
• Experience integrating data from databases, APIs, files, cloud storage, and enterprise applications.
• Strong experience working with relational databases such as SQL Server, Oracle, PostgreSQL, or MySQL.
• Experience with cloud platforms such as AWS and/or Microsoft Azure.
• Experience with Azure Data Factory, AWS Glue, EMR, or similar cloud data integration services.
• Experience working with cloud storage technologies such as Azure Data Lake Storage, Amazon S3, or equivalent.
• Experience with Apache Kafka or similar messaging and streaming technologies.
• Experience developing data pipelines using Apache Airflow or similar workflow orchestration tools.
• Strong understanding of data ingestion, transformation, validation, cleansing, and reconciliation processes.
• Experience implementing incremental data loads, CDC, and SCD Type 1/Type 2 methodologies.
• Strong understanding of data quality, data governance, data lineage, and data security principles.
• Experience optimizing Spark jobs, SQL queries, data pipelines, and large-scale data processing workloads.
• Experience working with large datasets and distributed computing environments.
• Experience with Git and source-code management practices.
• Experience implementing CI/CD pipelines for data engineering and data platform deployments.
• Experience with Jenkins, GitHub Actions, GitLab CI, Azure DevOps, or similar DevOps tools.
• Experience deploying and managing data engineering solutions across development, QA, and production environments.
• Experience working with APIs, JSON, XML, CSV, Parquet, Avro, and other structured/semi-structured data formats.
• Experience working with data ingestion from REST APIs and third-party systems.
• Strong understanding of data partitioning, indexing, file formats, and query optimization.
• Experience monitoring data pipelines and troubleshooting failed jobs, data issues, and production incidents.
• Experience implementing logging, monitoring, alerting, and operational support for data pipelines.
• Experience working with data security, access controls, encryption, and sensitive enterprise data.
• Experience collaborating with Data Architects, Software Engineers, BI Developers, Data Scientists, QA Engineers, DevOps Engineers, and Business Analysts.
• Experience participating in technical design discussions, code reviews, and architecture decisions.
• Strong understanding of SDLC, Agile/Scrum, testing, deployment, and production support processes.
• Ability to analyze complex business and technical requirements and translate them into scalable data solutions.
• Strong analytical and problem-solving skills.
• Excellent communication and collaboration skills.
• Ability to work independently while effectively collaborating with distributed technical teams.
• Strong experience delivering scalable, reliable, secure, and high-performance data engineering solutions.