Summary
Overview
Work History
Skills
Websites
Certification
Education
Timeline
Generic
SHIVAPRASAD P

SHIVAPRASAD P

Plano,TX

Summary

Certified Data Engineer with over 4+ years of hands-on experience and a proven track record in the design, development, and implementation of robust data engineering solutions. I am certified in Azure Data Engineer DP-203 and Azure Data Science DP-100. I bring expertise in ETL processes tailored to diverse business models. Adept at leveraging GenAI, analytical tools, and machine learning algorithms to drive data-driven insights and seeking a challenging role to apply my skills in a dynamic environment.

Overview

5
5
years of professional experience
5
5
Certifications

Work History

Data Engineer

almIT, DataPI
06.2023 - Current
  • Implemented Lakehouse Medallion Architecture in Azure Databricks, optimizing data management across raw, silver, and gold layers for improved accessibility and analysis
  • Developed and maintained PySpark notebooks for data validation and transformation, boosting accuracy and reliability
  • Engineered data processing workflows using sequential and parallel strategies in Azure Databricks, enhancing throughput and responsiveness for daily incremental loads
  • Parsed and structured unstructured data like JSON using Azure Stream Analytics
  • Managed telemetric data integrity through Event Hubs and processed data in Azure Databricks.
  • Implemented GenAI anomaly detection models to identify irregular patterns in telemetry, enhancing predictive capabilities
  • Applied Azure DevOps for agile methodology, improving team collaboration and project management
  • Conducted predictive analytics on historical HVAC data, aiding in proactive system management
  • Led an IoT project for monitoring environmental conditions using Azure Digital Twins and IoT edge devices, enhancing real-time data accuracy and system reliability and developed Python scripts for data integration from IoT devices to Azure Event Hubs via Azure Stream Analytics
  • Enhanced data visualization and reporting by integrating machine learning algorithms in Databricks, using Delta-Live Tables for high-velocity IoT data integration
  • Utilized Power BI for insightful visualizations and real-time reporting, integrating data from Azure Data Lake for immediate access
  • Managed Databricks workspaces and ensured cost efficiency through rigorous resource management.

Data Engineer

Agadia
Parsippany, NJ
01.2023 - 04.2023
  • Established interconnected services linking SQL Server, and Azure Synapse Analytics' dedicated SQL pool, creating a robust infrastructure for customer and claims database management.
  • Engineered and deployed efficient ETL pipelines across a variety of databases, significantly enhancing the team's capacity to scrutinize and refine service offerings.
  • Leveraged the transformative capabilities of Azure Data Factory, employing data flow activities like filtering, aggregation, and type conversion to elevate the caliber and applicability of data.
  • Coordinated the seamless transfer of data through various stages within the pipeline via Azure Data Factory, facilitating superior data consolidation and integration efforts.
  • Integrated Azure Event Hubs to gather and analyze real-time data from assorted sources, achieving a seamless connection with Azure Data Factory for efficient data capture.
  • Utilized Databricks to perform complex data transformations and load processed data efficiently into Azure Synapse Analytics' dedicated SQL pools, ensuring seamless data flows and integrity.
  • Determined critical data elements, established target schemas, and maintained data uniformity across the dedicated SQL pool, ensuring data integrity.
  • Adopted incremental loading tactics to refresh only newly altered or added data, thereby streamlining data updates and maintaining synchronization.
  • Employed change detection techniques to accurately track and update altered records, ensuring the freshness and accuracy of customer data.
  • Scheduled Azure Data Factory pipelines with precise triggers for automated runs, optimizing operations through event-based execution.

Student Assistant

University of Missouri Kansas City, FPHLM
08.2022 - 12.2022
  • Led automated reporting development using Python for predictive damage assessment in the Florida Public Hurricane Loss Model project
  • Conducted extensive Exploratory Data Analysis (EDA) using Python libraries and created visualizations with R to support decision-making
  • Managed server infrastructure and large datasets using SQL and developed custom reports with SQL Server Reporting Services (SSRS).

Data Engineer

GrayLogic Technologies, ATOM
10.2019 - 12.2021
  • Established interconnected services linking SQL Server with Azure Synapse Analytics' dedicated SQL pool, enhancing infrastructure for customer and claims database management
  • Engineered and deployed efficient ETL pipelines across multiple databases, boosting the team's ability to analyze and refine services, utilized Azure Data Factory for data manipulation, implementing activities like filtering, aggregation, and type conversion to improve data quality and relevance
  • Enforced PII and PCI compliance in data handling practices
  • Coordinated data transfer through Azure Data Factory, achieving enhanced data consolidation and integration
  • Integrated Azure Event Hubs for real-time data analysis, streamlining data capture and integration with Azure Data Factory
  • Used Databricks for complex data transformations, loading data efficiently into Azure Synapse Analytics, maintaining data flow and integrity
  • Scheduled Azure Data Factory pipelines with event-based triggers, optimizing operations and workflow automation
  • Collaborated with data analysts, using Power BI integrated with Azure technologies to provide a comprehensive view of essential business metrics such as claims, customer demographics, and churn rates.
  • Led the design and implementation of a web-based application for fee processing using Java Server Pages (JSP), Java, and JavaScript at Kakatiya University
  • Enhanced user engagement and academic auditing processes through continuous website upgrades and integration of advanced functionalities
  • Managed the MySQL database, ensuring data integrity and security, and collaborated with stakeholders for seamless payment integration with the State Bank of India (SBI)
  • Directed the entire web application lifecycle, ensuring alignment with university needs and strategic objectives.

Skills

Data Engineering: Azure Databricks, Azure Data Factory, Azure Digital Twins, PySpark, Azures Synapse Analytics, Azure ML Studio, Azure Stream Analytics, Azure IoT Hubundefined

Certification

Microsoft Certified: Azure Data Engineer Associate (DP-203)

Education

Master of Science - Computer Science

University of Missouri - Kansas City
Kansas City, MO

Bachelor of Technology - Computer Science And Engineering

Kakatiya University
Warangal

Timeline

Data Engineer

almIT, DataPI
06.2023 - Current

Data Engineer

Agadia
01.2023 - 04.2023

Student Assistant

University of Missouri Kansas City, FPHLM
08.2022 - 12.2022

Data Engineer

GrayLogic Technologies, ATOM
10.2019 - 12.2021

Master of Science - Computer Science

University of Missouri - Kansas City

Bachelor of Technology - Computer Science And Engineering

Kakatiya University

Similar Profiles

CREATE PROFILE
SHIVAPRASAD P