SQL Developer

TechDigital Group; United States; On-site; USD 80,000 - 120,000; Full time

Job summary

A leading company in the tech industry is seeking a Data Engineer to develop and maintain ETL/ELT pipelines using PySpark and SQL. The role involves working with both structured and unstructured data, optimizing data solutions, and collaborating with cross-functional teams. The ideal candidate will have strong experience in Python, SQL, and cloud platforms, ensuring high-quality data workflows. Join this innovative team to make a significant impact in data engineering.

Qualifications

• Strong experience in Python for data engineering tasks.
• Proficiency in PySpark for large-scale data processing.
• Deep understanding of SQL including optimization.

Responsibilities

• Develop, optimize, and maintain ETL/ELT pipelines using PySpark and SQL.
• Collaborate with Data Scientists and Analysts to integrate data workflows.
• Ensure data quality, validation, and consistency in data pipelines.

Skills

Python

SQL

PySpark

Tools

AWS

Azure

GCP

Airflow

Git

Job description

Mandatory Skills - SQL Server, Azure Devops
Responsibilities:

• Develop, optimize, and maintain ETL/ELT pipelines using PySpark and SQL.
• Work with structured and unstructured data to build scalable data solutions.
• Write efficient and scalable PySpark scripts for data transformation and processing.
• Optimize SQL queries, stored procedures, and indexing strategies to enhance performance.
• Design and implement data models, schemas, and partitioning strategies for large-scale datasets.
• Collaborate with Data Scientists, Analysts, and other Engineers to integrate data workflows.
• Ensure data quality, validation, and consistency in data pipelines.
• Implement error handling, logging, and monitoring for data pipelines.
• Work with cloud platforms (AWS, Azure, or GCP) for data processing and storage.
• Optimize data pipelines for cost efficiency and performance.

Technical Skills Required:

• Strong experience in Python for data engineering tasks.
• Proficiency in PySpark for large-scale data processing.
• Deep understanding of SQL (Joins, Window Functions, CTEs, Query Optimization).
• Experience in ETL/ELT development using Spark and SQL.
• Experience with cloud data services (AWS Glue, Databricks, Azure Synapse, GCP BigQuery).
• Familiarity with orchestration tools (Airflow, Apache Oozie).
• Experience with data warehousing (Snowflake, Redshift, BigQuery).
• Understanding of performance tuning in PySpark and SQL.
• Familiarity with version control (Git) and CI/CD pipelines.