IT professional with 4 years of experience specializing in Azure Cloud migration and development. Expertise in migrating SQL databases to Azure Data Lake, Azure SQL Database, and Azure SQL Data Warehouse. Skilled in developing Spark applications using Spark SQL for data extraction and transformation. Adept at designing complex application workloads on MS Azure.
Overview
9
9
years of professional experience
1
1
Certification
Work History
AZURE Data Engineer(
FeeL Inc(Stealth Startup)
, California, United States
02.2022 - Current
Applied expertise in comprehensive data integration solutions with Databricks, optimizing policyholder information transfer.
Played a pivotal role in data processing Avro, JSON, and XLSX formats using Delta Lake and applying advanced transformations for various calculations and risk profiling.
Leveraged ETL workflows using Synapse Pipelines and employed Spark jobs in Databricks for data processing.
Created data processing architecture utilizing Databricks with Spark for effective, high-volume dataset management.
AWS Data Engineer
Walmart
, AR
07.2020 - 11.2021
Extract Designed and developed Security Framework to provide fine grained access to objects in AWS S3 using AWS Lambda, AWS Glue DynamoDB
Performed end-to-end Architecture & implementation assessment of various AWS services like Amazon EMR, Redshift, S3
Used AWS EMR to transform and move large amounts of data into and out of other AWS data stores and databases, such as Amazon Simple Storage Service (Amazon S3) and Amazon DynamoDB
Import the data from different sources like HDFS/HBase into Spark RDD and perform computations using PySpark to generate the output response
Experience in implementing AWS lambda to run servers without managing them and to trigger run code by S3 & SNS
Worked on implementing Data warehouse solutions in AWS Redshift, worked on various projects to migrate data from one database to AWS Redshift, RDS, ELB, EMR, Dynamo DB and S3
Experience in AWS Glue for fully managed ETL service that makes it easy to move data between data stores and Manage ETL jobs that automate data processing and reporting
Worked on Spark for improving the performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark SQL and Spark Yarn
Developed Sqoop and Kafka Jobs to load data from RDBMS, External Systems into HDFS and HIVE
Developed Oozie coordinators to schedule Hive scripts to create Data pipelines
Written several Map reduce Jobs using Pyspark, Numpy and used Jenkins for Continuous integration
Setting up and worked on Kerberos authentication principals to establish secure network communication
On cluster and testing of HDFS, Hive, Pig and MapReduce to access cluster for new users
Continuous monitoring and managing the Hadoop cluster through Cloudera Manager
Python Developer/ETL Developer
MICRONOX PVT LTD
HYDERABAD, INDIA
08.2015 - 07.2018
Developed data integration strategies employing machine learning algorithms in Python and streamlined data visualization processes through custom Pandas tools.
Utilized Django and manual SQL for real-time model modifications, and also enhanced efficiency with code optimization and effective memory management.
Created solutions for JSON-based REST web services data extraction. Utilize Panda data frames and MySQL for dataset management.
Involved in Web/Application development using Python 3.5, HTML5, CSS3, AJAX, JSON, and jQuery.
Built various graphs for business decision making using the Python Matplotlib library.
Performed troubleshooting, fixed, and deployed many Python bug fixes of the two main applications that were a main source of data for both customers and the internal customer service team.
Education
Master of Science - Computer And Information Sciences
University of Bridgeport
Bridgeport, CT
04-2023
Skills
Data Migration
Data pipeline control
Data integration
Data Security
Data curating
Data Warehousing
ETL development
Data Modeling
Data Governance
Big data technologies
SQL and Databases
Database Design
Data Analysis
Certification
COMPLETED CERTIFICATION ON AZURE DATA ENGINEER ASSOCIATE (CERTIFIED MICROSOFT)
Achieved a certification on Machine learning by GUVI .
Timeline
AZURE Data Engineer(
FeeL Inc(Stealth Startup)
02.2022 - Current
AWS Data Engineer
Walmart
07.2020 - 11.2021
Python Developer/ETL Developer
MICRONOX PVT LTD
08.2015 - 07.2018
COMPLETED CERTIFICATION ON AZURE DATA ENGINEER ASSOCIATE (CERTIFIED MICROSOFT)
Achieved a certification on Machine learning by GUVI .
Master of Science - Computer And Information Sciences
University of Bridgeport
Similar Profiles
Abinash MishraAbinash Mishra
Building Advanced AI Products, Founder at Stealth AI Startup · Self-employedBuilding Advanced AI Products, Founder at Stealth AI Startup · Self-employed