Data Engineer - Python, SQL, Databricks

Insight Global (United States); Westbrook (ME); On-site; USD 140,000 - 190,000; Full time

Job summary

Insight Global is seeking a Senior Data Engineer to design, build, and optimize scalable data pipelines supporting analytics, AI, and ML initiatives. You will own pipeline development from ingestion through transformation and delivery while driving data quality, reliability, and engineering best practices.

You will design and implement complex SQL transformations, data models, and medallion-based architectures, and build DBT testing and validation frameworks.

Qualifications

• Advanced SQL and data modeling expertise.
• Strong knowledge of data warehousing and ETL/ELT architecture.
• Spark SQL and Python.
• dbt Core or dbt Cloud major plus.
• Streaming ingestion and schema evolution.
• Parquet and JSON processing.

Responsibilities

• Design and build scalable, fault-tolerant data pipelines using modern cloud data platforms.
• Develop complex SQL transformations and data models leveraging medallion architecture.
• Build and maintain DBT testing, data quality, and validation frameworks.
• Support batch and streaming ingestion, including schema evolution and Parquet/JSON processing.
• Optimize pipeline performance, reliability, and scalability.
• Manage CI/CD and deployment workflows through GitHub.
• Partner with ML engineers and data scientists to support AI/ML data and feature pipelines.
• Drive strong documentation, lineage, and engineering standards.

Job description

Job Description

Seeking a Senior Data Engineer to design, build, and optimize scalable data pipelines supporting analytics, AI, and machine learning initiatives. This role will own pipeline development from ingestion through transformation and delivery while driving data quality, reliability, and engineering best practices.

Key Responsibilities

• Design and build scalable, fault-tolerant data pipelines using modern cloud data platforms.
• Develop complex SQL transformations and data models leveraging medallion architecture.
• Build and maintain DBT testing, data quality, and validation frameworks.
• Support batch and streaming ingestion, including schema evolution and Parquet/JSON processing.
• Optimize pipeline performance, reliability, and scalability.
• Manage CI/CD and deployment workflows through GitHub.
• Partner with ML engineers and data scientists to support AI/ML data and feature pipelines.
• Drive strong documentation, lineage, and engineering standards.

Skills and Requirements

• Advanced SQL and strong data modeling expertise.
• Strong knowledge of data warehousing and ETL/ELT architecture.
• Experience with Databricks, Snowflake, AWS, or similar cloud data platforms.
• Experience with CI/CD, GitHub, and production data pipelines.
• 8+ years of data engineering experience.
• dbt Core or dbt Cloud major plus
• Spark SQL and Python
• Streaming ingestion and schema evolution
• Parquet and JSON processing

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal employment opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment without regard to race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or the recruiting process, please send a request to [e-mail].