Senior Data Systems Engineer – Big Data & ETL

VLink, Inc.; Chicago (IL); Hybrid; USD 125,000 - 180,000; Full time

Job summary

VLink, Inc. is seeking an experienced Data Engineer/Systems Analyst to design, build and support enterprise data solutions using Databricks, Spark, Python and Azure Data Factory.

You will collaborate with data architects, engineers and stakeholders across multiple domains, optimize ETL pipelines, and ensure data quality and governance in a large-scale cloud environment. This role demands 8+ years in data engineering, strong SQL, Hive, Hadoop, and scripting, with experience in agile teams and

Qualifications

• 8+ years of experience in Systems Analysis, Data Engineering, Big Data Development, ETL/ELT, or enterprise data integration.
• Strong hands on experience with Python, PySpark, Apache Spark, SQL, Databricks, and Azure Data Factory (ADF).
• Experience analyzing, designing, developing, and supporting enterprise scale data processing applications and ETL/ELT pipelines.
• Strong experience gathering business and technical requirements, performing system impact analysis, documenting functional specifications, and developing source to target data mappings.
• Hands on experience with Azure Data Factory (ADF) for data ingestion, transformation, integration, and workflow orchestration.
• Experience with Hadoop ecosystem technologies, including Hadoop/HDFS and Hive.
• Strong SQL skills, including complex queries, stored procedures, data transformations, reconciliation, and performance optimization.
• Experience developing, analyzing, and supporting ETL workflows using SSIS.
• Experience with Unix/Linux Shell Scripting and enterprise job scheduling tools such as Control M.
• Strong understanding of Delta Lake, Databricks Workflows, Lakehouse architecture, data warehousing, data modeling, and enterprise data integration.
• Experience with data profiling, data validation, source to target reconciliation, data quality frameworks, and enterprise data management practices.

Responsibilities

• Analyze, design, develop, and support enterprise data management solutions, Big Data platforms, ETL/ELT pipelines, and data integration processes.
• Collaborate with business stakeholders, data architects, and engineering teams to gather, analyze, and translate business requirements into functional and technical specifications.
• Perform system analysis, impact assessments, gap analysis, data mapping, and source to target transformation analysis across enterprise data platforms.
• Design, develop, and enhance large scale data processing solutions using Databricks, Apache Spark, PySpark, Python, Hadoop/HDFS, Hive, and Azure Data Factory (ADF).
• Analyze and support Databricks notebooks, Spark jobs, Delta Lake tables, and Lakehouse data processing workflows.
• Design and maintain data ingestion, transformation, and integration pipelines using Azure Data Factory (ADF), including activities, triggers, parameters, and dependencies.
• Develop and optimize complex SQL queries, stored procedures, data transformations, and source to target mapping specifications.
• Analyze, develop, and maintain ETL workflows using SSIS and modern cloud based data integration technologies.
• Develop and maintain reusable Python, PySpark, and Unix/Linux Shell Scripts for data processing, automation, validation, and operational support.
• Analyze data integration requirements across relational databases, APIs, files, cloud storage, and enterprise applications.
• Support batch processing, job scheduling, dependency management, and workflow orchestration using Control M, ADF, and Databricks Workflows.
• Perform data profiling, validation, reconciliation, root cause analysis, and troubleshooting to ensure data accuracy, completeness, and integrity.
• Analyze and optimize Spark jobs, Databricks workloads, SQL queries, and data pipelines for performance, scalability, and reliability.
• Prepare functional specifications, technical design documents, data mapping documents, process flows, and system integration documentation.
• Collaborate with developers, QA teams, Product Owners, Scrum Masters, and architects to support solution development, system testing, integration testing, and release activities.
• Participate in Agile/Scrum ceremonies, including sprint planning, backlog refinement, daily stand ups, reviews, and retrospectives.
• Support production deployments, incident investigation, defect resolution, change management, and post deployment validation to maintain enterprise data platform stability.

Skills

Python

PySpark

Apache Spark

SQL

Databricks

Azure Data Factory

Hadoop/HDFS

Hive

Shell Scripting

Control-M

Agile/Scrum

Stakeholder communication

Tools

SSIS

Tableau/Power BI

Kafka

Airflow

Git

Jenkins

Job description

VLink, Inc. is seeking an experienced Data Engineer/Systems Analyst to design, build and support enterprise data solutions using Databricks, Spark, Python and Azure Data Factory.

You will collaborate with data architects, engineers and stakeholders across multiple domains, optimize ETL pipelines, and ensure data quality and governance in a large-scale cloud environment. This role demands 8+ years in data engineering, strong SQL, Hive, Hadoop, and scripting, with experience in agile teams and