Varun Kumar Mishra
Berlin
Summary
Data Engineer with a demonstrated track record of working with leading Big Data ecosystem vendors like Cloudera, AWS. Skilled in Python, pyspark, AWS, Hive, SQL, Kafka. I have around 8 yrs. of experience working in Big data engineering. Having worked with data on petabyte-scale serving, streaming, and batch processing analytics has given immense exposure and the opportunity to implement and hone technical and cognitive skills.
Overview
8
8
years of professional experience
Work History
Senior Data Engineer
Bitcapital
01.2023 - Current
- Writing ETLs to ingest data from various sources like of data vendors like sftp,snowflake,s3,rest APIs.
- Maintaining and optimising Postgres(RDS) sql queries for Business requirements.
- scheduling and automated monitoring by sending alerts in case of job failure of data pipelines on Airflow.
- AWS lambda to trigger event based jobs.
- Design and implementation of AWS infra end to end for our internal DWH like creating EC2,Directory service, cost optimisation etc.
- Worked on vector databases like (Myscale/Pinnecone) to create embedding for creating retrieval augmented system.
- Creating APIs to share with end users using API gateways and flask.
Technical Lead
Paytm Payments Bank
02.2022 - 01.2023
- Created ETL workflows to ingest data from oracle golden gate kafka using pyspark streaming
- Created ETL pipeline on AWS cloud and in-house HDP cluster using python/pyspark and scheduling the workflows on Airflow
- Optimizing SQL queries and pyspark jobs
- Created reporting jobs in python using pandas and numpy
- Creating dashboards on BI Tools like Superset
- Worked on setting up superset on Docker
- Worked on creating an in-house DIY tool for end users to schedule hive queries.
- POC for new projects and creating SOPs(end-to-end from design/infra set up on docker to job scheduling)
- Leading a team of data engineers.
Customer Operation Engineer
Cloudera Inc.
08.2020 - 01.2022
- SME for YARN, SPARK
- Wrote, fine tuned and debugged pyspark and yarn applications
- Worked on identifying product BUG and providing workarounds and hotfixes/patches
- Created automation scripts for spark application monitoring using Python and Bash
Senior Software Engineer
Paytm Payments Bank
09.2019 - 08.2020
- Worked on setting up Paytm bank DWH cluster
- Created end user reporting jobs using pandas/numpy.
- writing pyspark ETLs and hive queries.
- optimizing hive queries.
- Yarn jobs resource utilization tuning and optimization.
- Setting up automated monitoring for the DWH.
- Security implementation of DWH using ranger.
Data Engineer
Impetus Pvt. Ltd.
06.2018 - 09.2019
- Kyvos is a product to provide Fast and scalable OLAP/BI solution on HDP,CDH,EMR clusters
- Created datasets, relationships, and cubes in kyvos
- Created Dashboards and visualizations on cubes to provide less than 10s response time to queries
- Integrated cubes with Tableau and Excel for BI and analytics.
Senior System Engineer
Infosys Pvt. Ltd.
02.2016 - 06.2018
- Created ETL pipelines using pyspark and python
- Created monitoring scripts for the pyspark jobs
- Working of monitoring and maintenance of Hadoop clusters on HDP.
Education
Bachelor of Technology in Electrical Engineering -
Haldia Institute of Technology
West Bengal, India06.2015
Skills
- Postgres SQL
- Python
- Data Modelling
- Hive
- Spark
- Hadoop(HDFS, YARN)
- Kafka
- AWS (S3,EC2,Lambda,API Gateway, ELB, Directory service, IAM, Athena)
- Data Warehousing
- Performance Tuning
- Docker
- Git
Languages
English
Bilingual or Proficient (C2)
German
Beginner (A1)
Timeline
Senior Data Engineer
Bitcapital
01.2023 - Current
Technical Lead
Paytm Payments Bank
02.2022 - 01.2023
Customer Operation Engineer
Cloudera Inc.
08.2020 - 01.2022
Senior Software Engineer
Paytm Payments Bank
09.2019 - 08.2020
Data Engineer
Impetus Pvt. Ltd.
06.2018 - 09.2019
Senior System Engineer
Infosys Pvt. Ltd.
02.2016 - 06.2018
Bachelor of Technology in Electrical Engineering -
Haldia Institute of Technology
Similar Profiles
Zahra El KhalloufiZahra El Khalloufi
Marketing Manager French Market at Gala Group BrandsMarketing Manager French Market at Gala Group Brands
Punith Nelamangala Shantha KumarPunith Nelamangala Shantha Kumar
Operational Excellence at Flink SEOperational Excellence at Flink SE
Martina FaleriMartina Faleri
Sales Development Representative (Level III) at NinjaOne GmbHSales Development Representative (Level III) at NinjaOne GmbH