Summary
Overview
Work History
Education
Skills
Websites
Certification
Timeline
Generic
SHIVAPRASAD PANASAM

SHIVAPRASAD PANASAM

Summary

Certified Data Engineer with over 4 years of hands-on experience and a demonstrated track record in the design, development, and implementation of robust data engineering solutions. Certified in Azure Data Engineer DP-203 and Azure Data Science DP-100, bringing expertise in ETL processes tailored to diverse business models. Adept at leveraging analytical tools, and machine learning algorithms to drive data-driven insights, seeking a challenging role to apply extensive skills in a dynamic environment.

Overview

5
5
years of professional experience
1
1
Certification

Work History

Data Engineer

almIT, DataPI
06.2023 - Current
  • Streamlined data management within Azure Databricks Lakehouse Medallion Architecture, optimizing accessibility and analytical readiness across raw, silver, and gold layers.
  • Developed and maintained PySpark notebooks for data validation and transformation, boosting accuracy and reliability.
  • Engineered advanced data processing workflows in Azure Databricks using sequential and parallel strategies, improving daily data load throughput and system responsiveness by 35%. Managed data loads involving over 6 million records daily across clusters scaling to 50 nodes.
  • Parsed and structured JSON data using Azure Stream Analytics, enhancing data usability for analytics.
  • Maintained telemetry data integrity from almIT Landing Zone Event Hubs to Staging Zone ADLS, ensuring 99% data accuracy for critical operations.
  • Developed PySpark notebooks for almIT Staging Zone ADLS, producing KPI-driven reports for almIT Azure Synapse Data Warehousing system, enhancing decision-making efficiency.
  • Implemented GenAI-based anomaly detection models, identifying critical data outliers, increasing anomaly detection accuracy.
  • Applied Azure DevOps for agile processes, enhancing team collaboration and project delivery speed.
  • Enhanced historical HVAC system data analytics, forecasting system behavior to prevent 15% of potential failures through predictive measures.
  • Led an IoT project for monitoring environmental conditions using Azure Digital Twins and IoT edge devices, enhancing real-time data accuracy and system reliability and developed Python scripts for data integration from IoT devices to
    Azure Event Hubs via Azure Stream Analytics.
  • Developed Python scripts and utilized Azure Stream Analytics for efficient data transfer from IoT edge devices to Azure Event Hubs, ensuring integrity and timely availability of telemetric data in Databricks for further processing.
  • Generated actionable insights using Power BI with data from Azure Data Lake, improving real-time decision-making capabilities.
  • Utilized Databricks’ Delta-Live Tables for the continuous integration of sensor data, enabling real-time data feeds and streamlining the data pipeline for high-velocity IoT data.
  • Integrated ML models into IoT edge devices, showcasing expertise in AI, machine learning, and IoT

Data Engineer

Agadia, ATLAS
01.2023 - 05.2023
  • Established interconnected services linking SQL Server, and Azure Synapse Analytics' dedicated SQL pool, creating a robust infrastructure for customer and claims database management.
  • Engineered and deployed efficient ETL pipelines across a variety of databases, significantly enhancing the team's capacity to scrutinize and refine service offerings.
  • Leveraged the transformative capabilities of Azure Data Factory, employing data flow activities like filtering, aggregation, and type conversion to elevate the caliber and applicability of data.
  • Coordinated the seamless transfer of data through various stages within the pipeline via Azure Data Factory, facilitating superior data consolidation and integration efforts.
  • Integrated Azure Event Hubs to gather and analyze real-time data from assorted sources, achieving a seamless connection with Azure Data Factory for efficient data capture.
  • Utilized Databricks to perform complex data transformations and load processed data efficiently into Azure Synapse Analytics' dedicated SQL pools, ensuring seamless data flows and integrity.
  • Determined critical data elements, established target schemas, and maintained data uniformity across the dedicated SQL pool, ensuring data integrity.
  • Adopted incremental loading tactics to refresh only newly altered or added data, thereby streamlining data updates and maintaining synchronization.
  • Employed change detection techniques to accurately track and update altered records, ensuring the freshness and accuracy of customer data.
  • Scheduled Azure Data Factory pipelines with precise triggers for automated runs, optimizing operations through event-based execution.

Student Assistant

University of Missouri Kansas City, FPHLM
08.2022 - 12.2022
  • Led automated reporting development using Python for predictive damage assessment in the Florida Public Hurricane Loss Model project
  • Conducted extensive Exploratory Data Analysis (EDA) using Python libraries and created visualizations with R to support decision-making
  • Managed server infrastructure and large datasets using SQL and developed custom reports with SQL Server Reporting Services (SSRS).

Data Engineer

GrayLogic Technologies, ATOM
10.2019 - 12.2021
  • Led development of a comprehensive data integration solution using Microsoft Azure services.
  • Engineered an end-to-end data pipeline for real-time consolidation and processing of diverse data
    sources.
  • Provided oversight to the daily operations, including data pulls, research assignments, file transfers, data
    processing and data quality management
  • Enforced PII and PCI compliance in data handling practices
  • Coordinated data transfer through Azure Data Factory, achieving enhanced data consolidation and integration
  • Integrated Azure Event Hubs for real-time data analysis, streamlining data capture and integration with Azure Data Factory
  • Used Databricks for complex data transformations, loading data efficiently into Azure Synapse Analytics, maintaining data flow and integrity
  • Scheduled Azure Data Factory pipelines with event-based triggers, optimizing operations and workflow automation
  • Collaborated with data analysts, using Power BI integrated with Azure technologies to provide a comprehensive view of essential business metrics such as claims, customer demographics, and churn rates.
  • Led the design and implementation of a web-based application for fee processing using Java Server Pages (JSP), Java, and JavaScript at Kakatiya University
  • Enhanced user engagement and academic auditing processes through continuous website upgrades and integration of advanced functionalities
  • Managed the MySQL database, ensuring data integrity and security, and collaborated with stakeholders for seamless payment integration with the State Bank of India (SBI)
  • Directed the entire web application lifecycle, ensuring alignment with university needs and strategic objectives.

Education

Master of Science - Computer Science

University of Missouri - Kansas City
Kansas City, MO

Bachelor of Technology - Computer Science And Engineering

Kakatiya University
India

Skills

  • Data Tools: Azure Databricks, Azure Data Factory, Azure Synapse Analytics, Azure Key Vault, Azure Logic Apps, Azure ML Studio
  • Data Integration: ETL, Data Warehousing, Data Lake house
  • Programming Languages: Python, SQL, C#, Java, C
  • Data Analysis: PySpark, Power BI, SSRS (SQL Server Reporting Services), Qlik Sense
  • Data Processing: Data Pipelines, Azure Stream Analytics
  • Machine Learning: Linear Regression, Logistic Regression, Naïve Bayes, Support Vector Machines
  • Exploratory Data Analysis (EDA): Data visualization, Statistical analysis, Pattern recognition
  • Feature Engineering: Feature selection, Feature scaling, Data transformation

Certification

  • Microsoft Certified: Azure Data Engineer Associate (DP-203)
  • Microsoft Certified: Azure Data Scientist Associate (DP-100)
  • Microsoft Certified: Azure Data Fundamentals (DP-900)
  • Microsoft Certified: Azure Fundamentals (AZ-900)
  • Python 101 for Data Science

Timeline

Data Engineer

almIT, DataPI
06.2023 - Current

Data Engineer

Agadia, ATLAS
01.2023 - 05.2023

Student Assistant

University of Missouri Kansas City, FPHLM
08.2022 - 12.2022

Data Engineer

GrayLogic Technologies, ATOM
10.2019 - 12.2021

Master of Science - Computer Science

University of Missouri - Kansas City

Bachelor of Technology - Computer Science And Engineering

Kakatiya University

Similar Profiles

CREATE PROFILE
SHIVAPRASAD PANASAM