Data Scientist II - Big Data Engineer

Cogent IBS, Inc; United States; Remote; USD 120,000 - 150,000; Full time

Job summary

Cogent IBS, Inc. seeks a Data Scientist (Big Data Engineer) II to design and optimize scalable data pipelines with Apache Spark on Databricks, and to integrate with Azure services.

You will own ETL/ELT workflows, data models, governance, security, and CI/CD deployments, collaborating with data scientists and analysts in Agile, multicultural teams. Proficiency in Python and SQL, plus Azure Data Lake/Delta Lake, is required.

Qualifications

• 4+ years ETL/ELT workflows for structured and unstructured data.
• 4+ years automating deployments using CI/CD tools.
• 4+ years collaborating with data scientists, analysts, stakeholders, and cross-functional teams.
• 4+ years designing and maintaining data models, schemas, and database structures.
• 4+ years working with data storage solutions, including Azure Data Lake Storage and data warehouses.
• 4+ years implementing data validation and data quality checks.
• 4+ years contributing to data governance, metadata management, data lineage, and data cataloging.
• 4+ years implementing data security measures, including encryption, access controls, and auditing.
• 4+ years of proficiency in Python and R programming languages.
• 4+ years of strong SQL querying and data manipulation experience.
• 4+ years of experience with the Microsoft Azure cloud platform.
• 4+ years of experience with DevOps, CI/CD pipelines, and version control systems.
• 4+ years working in Agile and multicultural environments.
• 4+ years of strong troubleshooting and debugging capabilities.
• 3+ years designing and developing scalable data pipelines using Apache Spark on Databricks.
• 3+ years optimizing Spark jobs for performance and cost efficiency.
• 3+ years integrating Databricks with Azure Data Factory.
• 3+ years ensuring data quality, governance, and security using Unity Catalog or Delta Lake.
• 3+ years of strong understanding of Apache Spark architecture, RDDs, DataFrames, and Spark SQL.
• 3+ years of hands-on experience with Databricks notebooks, clusters, jobs, and Delta Lake.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Apache Spark on Databricks.
• Implement ETL/ELT workflows for structured and unstructured data.
• Develop and optimize Spark jobs for performance and cost efficiency.
• Build and maintain data models, schemas, and database structures supporting analytical and operational use cases.
• Integrate Databricks solutions with Azure Data Factory and other Azure cloud services.
• Work with Azure Data Lake Storage and data warehouse solutions.
• Implement data validation and quality checks to ensure data accuracy, consistency, and reliability.
• Contribute to data governance initiatives, including metadata management, data lineage, and data cataloging.
• Implement data security measures, including encryption, access controls, and auditing.
• Support compliance with applicable regulations, security requirements, and industry best practices.
• Automate deployments using CI/CD pipelines, DevOps practices, and version control systems.
• Work with Databricks notebooks, clusters, jobs, and Delta Lake.
• Utilize Unity Catalog and/or Delta Lake to support data quality, governance, and security.
• Troubleshoot and debug data pipelines, Spark applications, and related technical issues.
• Collaborate with data scientists, data analysts, stakeholders, and cross-functional teams.
• Work effectively within Agile and multicultural environments.

Skills

Python

SQL

Spark

Databricks

Azure

CI/CD

Data Modeling

Delta Lake

Data Governance

Unix/Linux

Tools

Azure Data Factory

Azure Data Lake Storage

Delta Lake

Unity Catalog

Job description

ONLY LOCAL TO TEXAS

Data Scientist (Big Data Engineer) II -- Databricks / Azure
Position: Data Scientist (Big Data Engineer) II
Openings: 2
Location: 100% Remote
Duration: 12 Months ( Up to 3 years extension )

Rate : $65/Hr on C2C

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Apache Spark on Databricks.
• Implement ETL/ELT workflows for structured and unstructured data.
• Develop and optimize Spark jobs for performance and cost efficiency.
• Build and maintain data models, schemas, and database structures supporting analytical and operational use cases.
• Integrate Databricks solutions with Azure Data Factory and other Azure cloud services.
• Work with Azure Data Lake Storage and data warehouse solutions.
• Implement data validation and quality checks to ensure data accuracy, consistency, and reliability.
• Contribute to data governance initiatives, including metadata management, data lineage, and data cataloging.
• Implement data security measures, including encryption, access controls, and auditing.
• Support compliance with applicable regulations, security requirements, and industry best practices.
• Automate deployments using CI/CD pipelines, DevOps practices, and version control systems.
• Work with Databricks notebooks, clusters, jobs, and Delta Lake.
• Utilize Unity Catalog and/or Delta Lake to support data quality, governance, and security.
• Troubleshoot and debug data pipelines, Spark applications, and related technical issues.
• Collaborate with data scientists, data analysts, stakeholders, and cross-functional teams.
• Work effectively within Agile and multicultural environments.

Required Qualifications

• 4+ years of experience implementing ETL/ELT workflows for structured and unstructured data.
• 4+ years of experience automating deployments using CI/CD tools.
• 4+ years collaborating with data scientists, analysts, stakeholders, and cross-functional teams.
• 4+ years designing and maintaining data models, schemas, and database structures.
• 4+ years working with data storage solutions, including Azure Data Lake Storage and data warehouses.
• 4+ years implementing data validation and data quality checks.
• 4+ years contributing to data governance, metadata management, data lineage, and data cataloging.
• 4+ years implementing data security measures, including encryption, access controls, and auditing.
• 4+ years of proficiency in Python and R programming languages.
• 4+ years of strong SQL querying and data manipulation experience.
• 4+ years of experience with the Microsoft Azure cloud platform.
• 4+ years of experience with DevOps, CI/CD pipelines, and version control systems.
• 4+ years working in Agile and multicultural environments.
• 4+ years of strong troubleshooting and debugging capabilities.
• 3+ years designing and developing scalable data pipelines using Apache Spark on Databricks.
• 3+ years optimizing Spark jobs for performance and cost efficiency.
• 3+ years integrating Databricks with Azure Data Factory.
• 3+ years ensuring data quality, governance, and security using Unity Catalog or Delta Lake.
• 3+ years of strong understanding of Apache Spark architecture, RDDs, DataFrames, and Spark SQL.
• 3+ years of hands-on experience with Databricks notebooks, clusters, jobs, and Delta Lake.

Preferred Qualifications

• Knowledge of machine learning libraries such as:

• MLflow

• Scikit-learn

• TensorFlow

• Databricks Certified Associate Developer for Apache Spark certification.

• Microsoft Certified: Azure Data Engineer Associate certification.

Role Overview:
The position involves designing, developing, and optimizing scalable data pipelines and big data solutions using Azure, Databricks, and Apache Spark. The role also involves data quality, governance, security, CI/CD, and collaboration with data engineering, analytics, and business teams.