Senior Data Engineer - Databricks & ETL (1-Year)

InterviewPanel; United States; Remote; USD 110,000 - 170,000; Full time

Job summary

InterviewPanel is seeking a data integration and reporting expert to design, develop, and maintain robust ETL pipelines on Databricks. You will troubleshoot performance issues, migrate SSIS workflows to Databricks, and support data warehousing and reporting across the organization.

The role requires strong SQL Server skills (T-SQL, procedures) and extensive Databricks experience with Python, Spark, and Delta Lake. Excellent communication and collaboration abilities are essential.

Qualifications

• Bachelor's degree in computer science, information systems, or related field.
• 7+ years of experience in data integration and reporting.
• Extensive experience with Databricks, including Python, Spark, and Delta Lake.
• Strong proficiency in SQL Server, including T-SQL, stored procedures, and functions.
• Experience with SSIS development and maintenance.
• Experience with SSRS report design and development.
• Experience with data warehousing concepts and best practices.
• Experience with Microsoft Azure cloud platform and Microsoft Fabric desirable.
• Strong analytical and problem-solving skills.
• Excellent communication and interpersonal skills.
• Ability to work independently and as part of a team.
• Experience with Agile methodologies.

Responsibilities

• Design, develop, and maintain ETL pipelines and processes on Databricks.
• Troubleshoot Databricks pipeline errors and performance issues.
• Maintain legacy SSIS packages for ETL processes.
• Troubleshoot SSIS package errors and performance issues.
• Optimize data flow performance and minimize data latency.
• Implement data quality checks and validations within ETL processes.
• Develop and maintain Databricks pipelines and datasets using Python, Spark and SQL.
• Migrate legacy SSIS packages to Databricks pipelines.
• Optimize Databricks jobs for performance and cost-effectiveness.
• Integrate Databricks with other data sources and systems.
• Participate in the design and implementation of data lake architectures.
• Participate in the design and implementation of data warehousing solutions.
• Support data quality initiatives and implement data cleansing procedures.
• Collaborate with business users to understand data requirements for reporting needs.
• Maintain existing SSRS reports, dashboards, and visualizations.
• Troubleshoot and resolve SSRS report issues, including performance bottlenecks and data inconsistencies.

Skills

Data integration

Data reporting

Databricks

Python

Spark

SQL

T-SQL

Agile

Communication

Education

Bachelor's degree in Computer Science or related field

Tools

SSIS

SSRS

Databricks

Delta Lake

Azure

Job description

InterviewPanel is seeking a data integration and reporting expert to design, develop, and maintain robust ETL pipelines on Databricks. You will troubleshoot performance issues, migrate SSIS workflows to Databricks, and support data warehousing and reporting across the organization.

The role requires strong SQL Server skills (T-SQL, procedures) and extensive Databricks experience with Python, Spark, and Delta Lake. Excellent communication and collaboration abilities are essential.