Data Engineer

compoundexpress; Mumbai; Hybrid; INR 1,200,000 - 1,800,000; Full time

Job summary

compoundexpress is seeking a Data Engineer to own data pipelines in AWS and enable Data & Decision Science for product owners, marketing and senior leadership.

The role requires expertise in Python, Spark, SQL, and AWS-based big data technologies to build scalable data lakes, pipelines and analytics platforms.

Qualifications

• Education: B.Tech., Masters or MBA in Business Analytics or Computer Engineering or related field.
• Experience: 2 to 12 years; startup or financial services background preferred.
• Strong Python, Spark and SQL skills with relational and NoSQL databases.
• Experience building AWS/cloud data pipelines, architectures, and data sets.
• Experience with data transformation, metadata, workload management and pipelines.
• Knowledge of messaging/stream processing and scalable data stores.
• Familiarity with AWS services: EC2, EMR, Glue, Redshift, Sagemaker, Athena, QuickSight.

Responsibilities

• Assemble large, complex data sets into a Data Lake meeting business requirements.
• Design and implement ETL pipelines; optimize for streaming and batch data.
• Build infrastructure for data extraction, transformation using SQL, Spark, Python, Airflow and AWS.
• Ensure data security and governance across regions per infosec guidelines.
• Create analytics tools to aid product, marketing and leadership teams.
• Deliver scalable pipelines and robust architecture meeting business needs.
• Own data quality, correctness, validation, consistency, and reliability.
• Document and maintain production code for root-cause analysis.

Skills

Python

Spark

SQL

AWS

Data pipelines

Cross-functional collaboration

Education

B.Tech./ Masters/ MBA in Business Analytics or Computer Engineering or related field

Tools

Airflow

Azkaban

Luigi

Hadoop

Kafka

Postgres

Cassandra

EMR

Redshift

Sagemaker

Kinesis

Job description

Data Engineer

• Department: Data Analytics
• Location: Mumbai, Gurgaon, hybrid
• Positions: 4

What Is This Job About

Data Engineer will own setting up Data Pipeline in AWS environment and will enable Data & Decision Science for Products Owners, Marketing and Senior Leadership.

Responsibilities

• Assemble large, complex data sets from distinct sources into Data Lake that meet functional / non-functional business requirements.
• Identify, design, and implement data and ETL pipeline - optimizing data delivery, re-designing infrastructure for greater scalability, etc for both Stream and Batch data processing
• Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL, Spark, Python, Airflow and AWS ‘big data’ technologies.
• Keep our data separated and secure by adherence to infosec and regulatory guidelines related to data centers, governance, monitoring usage and selection of AWS regions.
• Create data & analytics tools/platform that assist Data & Analytics team members in building and optimizing our product into an innovative industry leader
• Expected OUTCOME : Deliver high quality Data Engineering Pipeline and Solution Architecture based on the business requirement which is scalable and streamlined
• Expected OUTCOME : Own data quality, correctness, validation, consistency & Reliability across pipeline
• Expected OUTCOME : Optimization of infrastructure based on usage and requirement
• Expected OUTCOME : Documentation and maintenance of Production code for efficient root cause analysis in case of any failures/issues

Key Skills

• EDUCATION Qualifications : B.Tech./ Masters/ MBA Degree in Business Analytics or Computer Engineering or related field
• EXPERIENCE : min. 2years upto 12years. Preference given to candidates who have worked in Startups and/or Financial Services domain.
• Advanced working Python, Spark & SQL knowledge and experience working with relational/non-relational databases
• Experience building and optimizing AWS/Cloud ‘big data’ data pipelines, architectures, solutions and data sets.
• Build processes supporting data transformation, data structures, metadata, dependency and workload management.
• Working knowledge of message queuing, stream processing, and highly scalable ‘big data’ data stores
• Experience supporting and working with cross-functional teams in a dynamic environment
• Experience with big data tools: Hadoop, Spark, Kafka, etc
• Experience with relational SQL and No SQL databases, including RDS, Postgres and Cassandra
• Experience with data pipeline and workflow management tools: Azkaban, Luigi, Airflow, etc.
• Experience with AWS cloud services: EC2, EMR, Batch, Lambda, RDS, Redshift, Sagemaker, AWS Glue, Athena, QuickSight, Kibana, ElasticSearch
• Experience with stream-processing systems: Storm, AWS Kinesis, Spark-Streaming, etc.