Senior Data Engineer: Python, SQL, Databricks & ML Pipelines

Insight Global (United States); Westbrook (ME); On-site; USD 140,000 - 190,000; Full time

Job summary

Insight Global is seeking a Senior Data Engineer to design, build, and optimize scalable data pipelines supporting analytics, AI, and ML initiatives. You will own pipeline development from ingestion through transformation and delivery while driving data quality, reliability, and engineering best practices.

You will design and implement complex SQL transformations, data models, and medallion-based architectures, and build DBT testing and validation frameworks.

Qualifications

• Advanced SQL and data modeling expertise.
• Strong knowledge of data warehousing and ETL/ELT architecture.
• Spark SQL and Python.
• dbt Core or dbt Cloud major plus.
• Streaming ingestion and schema evolution.
• Parquet and JSON processing.

Responsibilities

• Design and build scalable, fault-tolerant data pipelines using modern cloud data platforms.
• Develop complex SQL transformations and data models leveraging medallion architecture.
• Build and maintain DBT testing, data quality, and validation frameworks.
• Support batch and streaming ingestion, including schema evolution and Parquet/JSON processing.
• Optimize pipeline performance, reliability, and scalability.
• Manage CI/CD and deployment workflows through GitHub.
• Partner with ML engineers and data scientists to support AI/ML data and feature pipelines.
• Drive strong documentation, lineage, and engineering standards.

Job description

Insight Global is seeking a Senior Data Engineer to design, build, and optimize scalable data pipelines supporting analytics, AI, and ML initiatives. You will own pipeline development from ingestion through transformation and delivery while driving data quality, reliability, and engineering best practices.

You will design and implement complex SQL transformations, data models, and medallion-based architectures, and build DBT testing and validation frameworks.