Senior Data Engineer - AWS, Databricks & AI Pipelines

Tiger Analytics; Chicago, Northern (IL, KY); Hybrid; USD 140,000 - 190,000; Full time

Job summary

Tiger Analytics seeks an experienced Data Engineer to design, build and maintain scalable data pipelines on AWS, collaborating with data scientists and AI teams. You'll work with Databricks, Spark, and SQL to support analytics, machine learning, and Generative AI initiatives, while ensuring data quality and cost efficiency across pharma data sources and lakehouse architectures.

Significant career development opportunities exist as the company grows, offering a challenging, entrepreneurial

Qualifications

• 8+ years of experience in Data Engineering.
• Strong hands-on experience with AWS cloud, Databricks, Spark, and SQL.
• Strong experience building ETL/ELT data pipelines and large-scale data processing workflows.
• Hands-on experience with Apache Airflow for workflow orchestration.
• Strong understanding of data modeling, data lake/lakehouse architecture, data ingestion, and transformation frameworks.
• Deep knowledge of pharmaceutical data sources: Xponent, Veeva, MMIT, Plantrak, Specialty Pharmacy, LAAD, etc.
• Strong understanding of pharma KPIs, metrics, and commercial analytics.
• Strong analytical, problem-solving skills.

Responsibilities

• Design, develop, and deploy end-to-end data pipelines on AWS using services such as S3, Glue, Lambda, Redshift.
• Build and maintain scalable data processing and transformation workflows using Databricks, Spark, and SQL.
• Develop and maintain Apache Airflow workflows for orchestration, scheduling, monitoring, and automation.
• Integrate and process pharmaceutical data sources such as Xponent, Veeva, MMIT, Plantrak, Specialty Pharmacy, LAAD, and similar sources.
• Build and optimize data pipelines supporting pharma KPIs, analytics and reporting requirements.
• Design and implement data pipelines for AI/ML and Generative AI workloads, including data prep for structured and unstructured data.
• Enable pipelines for LLM-based applications, vector embeddings, and knowledge retrieval/RAG solutions.
• Support migration of legacy data systems and pipelines to modern AWS lakehouse architectures.
• Monitor, troubleshoot, and optimize data pipelines for performance, scalability, reliability, and cost-effectiveness.
• Ensure data pipelines meet data quality, accuracy, consistency, and operational reliability standards.
• Communicate effectively with technical and business stakeholders to translate pharma business needs into scalable data solutions.

Skills

AWS Cloud

Databricks

Spark

SQL

Airflow

Data Modeling

Data Lakehouse

ETL/ELT

Pharma data sources

Tools

AWS Glue

Redshift

Apache Airflow

Job description

Tiger Analytics seeks an experienced Data Engineer to design, build and maintain scalable data pipelines on AWS, collaborating with data scientists and AI teams. You'll work with Databricks, Spark, and SQL to support analytics, machine learning, and Generative AI initiatives, while ensuring data quality and cost efficiency across pharma data sources and lakehouse architectures.

Significant career development opportunities exist as the company grows, offering a challenging, entrepreneurial