Data Support Engineer

247Hire; Chicago (IL); On-site; USD 80,000 - 110,000; Full time

Job summary

247Hire is looking for a Data Application Support specialist who will focus on maintaining and optimizing data pipelines and Lakehouse architectures in Databricks. You'll resolve production incidents, tune SQL/PySpark jobs, manage Delta Lake assets, and collaborate with data engineering teams.

The ideal candidate will have strong experience in Apache Spark, Python, and SQL, along with proficiency in Databricks and Azure Data Lake technologies. If you excel in problem-solving and automation, this is the job for you!

Qualifications

• Strong hands-on experience with Apache Spark, Python/PySpark, and SQL.
• Proficiency with Databricks Unified Data Analytics Platform and Delta Lake.
• Experience with Azure Data Lake Storage (ADLS Gen2) or similar technologies.
• Experience with Git/Azure DevOps for code management.
• Strong analytical skills for diagnosing complex data issues.

Responsibilities

• Provide L2/L3 support for data applications, resolving production issues.
• Optimize Spark applications, SQL queries, and Delta Lake tables.
• Monitor and troubleshoot ETL/ELT pipelines in Databricks.
• Act as a liaison between users and data engineering teams.
• Create tools to automate routine support tasks.

Skills

Apache Spark

Python/PySpark

SQL

Databricks Unified Data Analytics Platform

Delta Lake

Azure Data Lake Storage (ADLS Gen2)

Git/Azure DevOps

Data Orchestration Tools

Data Governance Tools

Job description

A Data Application Support role specializing in Databricks focuses on maintaining, troubleshooting, and optimizing Spark-based data pipelines and Lakehouse architectures. Key responsibilities include resolving L2/L3 production incidents, performance tuning SQL/Python (PySpark) jobs, managing Delta Lake assets, and collaborating with data engineers to ensure data reliability.

Key Responsibilities

• Incident & Problem Management: Provide L2/L3 support for data applications, resolving production issues and troubleshooting Databricks jobs, notebooks, and workflows.
• Performance Tuning: Optimize Spark applications, SQL queries, and Delta Lake tables to improve efficiency and reduce costs.
• Pipeline Maintenance: Monitor and troubleshoot ETL/ELT pipelines in Databricks (including DataFactory/Delta Live Tables), ensuring data quality and lineage (UnityCatalog).
• Collaboration: Act as a liaison between users, data engineering teams, and platform engineering, providing technical expertise and contributing to documentation.
• Automation: Create tools to automate routine support tasks and enhance support team productivity.

Required Qualifications & Skills

• Technical Expertise: Strong hands-on experience with Apache Spark, Python/PySpark, and SQL.
• Databricks Ecosystem: Proficiency with Databricks Unified Data Analytics Platform, Delta Lake, and ideally Azure Databricks.
• Cloud Data Storage: Experience with Azure Data Lake Storage (ADLS Gen2) or similar data lake technologies.
• Version Control & CI/CD: Experience with Git/Azure DevOps for code management.
• Problem-Solving: Strong analytical skills, with the ability to diagnose complex data processing bottlenecks.

Nice-to-Have Skills

• Data orchestration tools (e.g., Apache Airflow, Azure Data Factory).
• Data governance tools (e.g., Unity Catalog, Collibra).
• Streaming data knowledge (e.g., Spark Structured Streaming).