Venkata P

Project: ICREMS(Intelligent Credit Risk Evaluation & Monitoring System)

• Designed Databricks Lakehouse architecture using Gold, Silver layers to ingest, curate, and serve enterprise credit-risk datasets at scale.
• Built Apache Spark batch pipelines for borrower, exposure, and transaction data with optimized partitioning, caching, and AQE.
• Implemented Delta Lake tables with ACID transactions, schema evolution, and time-travel for regulatory audit and replay use cases.
• Engineered ingestion frameworks conceptually aligned with Delta Live Tables (DLT) including SCD handling, data quality checks, and SLA monitoring.
• Migrated legacy Oracle and Cassandra credit datasets into a cloud-native Lakehouse backed by AWS S3 and Delta formats.
• Designed near real-time Spark Structured Streaming pipelines consuming Kafka events for continuous credit monitoring and alerting.
• Applied Unity Catalog–style governance patterns including RBAC, column-level security, and PII masking across curated datasets.
• Optimized Spark workloads through file compaction, Z-Ordering, and adaptive query execution, improving analytical query performance.
• Developed Databricks SQL–ready analytical views for portfolio risk, delinquency trends, and exposure analytics consumed by BI tools.
• Integrated MLflow-compatible experiment tracking for LLM-assisted borrower scoring and risk classification workflows.
• Implemented cloud security and IAM integration using AWS IAM roles, encryption at rest/in transit, and audit logging.
• Acted as data architecture advisor, guiding stakeholders on Lakehouse adoption, migration strategy, cost optimization, and scalability.

Project: Digital Prior Authorization Platform (DPAP)

• Designed Lakehouse-style data architecture to process clinical, pharmacy, and claims data across raw, curated, and analytics zones.
• Built PySpark ETL pipelines to ingest high-volume healthcare datasets into Delta Lake on Azure Data Lake Gen2.
• Implemented batch and micro-batch processing for near real-time authorization workflows using Spark SQL.
• Applied Dimensional Modeling to curated healthcare datasets supporting regulatory and operational analytics.
• Modernized legacy workflows by migrating from traditional batch systems to Azure Databricks–based pipelines.
• Implemented data quality rules, schema validation, and lineage tracking to support HIPAA-compliant governance.
• Designed secure access controls using RBAC, encryption, and integration with Azure AD .
• Built Databricks SQL–ready datasets and views powering compliance dashboards and utilization reporting.
• Collaborated on cloud migration strategy for moving on-prem healthcare data platforms to Azure Databricks Lakehouse.
• Optimized Spark jobs using partition pruning, caching, and efficient join strategies to reduce processing latency.
• Enabled downstream BI and ML workloads by publishing standardized, governed Delta tables.
• Partnered with enterprise data teams to define Lakehouse governance standards, cost controls, and operational best practices.

Project: Smart Promotions Engine

• Designed event-driven data ingestion pipelines producing Databricks-ready JSON and Parquet datasets for analytics use cases.
• Built scalable product and transaction data models aligned with Lakehouse and Delta modeling principles.
• Optimized Elasticsearch-backed data pipelines supporting analytics on 10M+ product records.
• Implemented batch-friendly and streaming-compatible schemas to support downstream Spark SQL analytics.
• Designed cloud-native data services deployed on Kubernetes, enabling scalable data ingestion and processing.
• Partnered with data engineering teams to publish curated datasets for growth analytics and forecasting.
• Applied performance tuning techniques across data ingestion, indexing, and query execution paths.
• Supported analytics and ML consumers by standardizing data contracts and event schemas