Lavanya Aalampally

Rosemount,MN

Summary

Senior Data Engineer with 10+ years of experience designing scalable data platforms and distributed systems in cloud-native environments. Expertise in PySpark, SQL, and real-time data processing, with a proven track record of migrating complex on-premise systems to Azure, Snowflake, and Databricks—delivering up to 75% cost reduction and 60% performance improvement. Experienced in building enterprise data lakehouse architectures, streaming pipelines (Kafka), and governed data platforms. Strong track record of leading end-to-end data engineering initiatives and translating business requirements into high-impact, data-driven solutions across healthcare and financial domains.

Overview

years of professional experience

Certification

Work History

Lead Data Engineer

Optum

Minneapolis, MN

09.2019 - Current

Architected and deployed a cloud-native analytics platform on Azure Databricks to process large-scale claims data using PySpark, improving performance by 60% and reducing costs by 75%.
Designed scalable, on-demand compute architecture and implemented Delta Lake for ACID-compliant, high-reliability data pipelines on ADLS Gen2 and Snowflake.
Built real-time data ingestion pipelines using Kafka and Python, reducing data latency from batch to near real-time.
Developed and deployed microservices on Kubernetes to support high-concurrency workloads with enhanced system resilience.
Led integration of external healthcare datasets, standardizing data models and enabling seamless enterprise-wide data access.
Implemented data governance, lineage, and access control using Unity Catalog to meet compliance requirements.
Designed automated data reconciliation frameworks using Python and Apache Airflow, eliminating manual validation efforts.
Created Power BI dashboards to monitor data quality, pipeline health, and operational metrics in real time.
Modernized legacy ETL pipelines by refactoring IBM DataStage workflows into optimized Teradata processes.
Orchestrated enterprise pipelines using Apache Airflow, ensuring reliability, scalability, and fault tolerance.
Led migration of sensitive customer and member data (PII) to modern platforms with zero downtime and zero data loss.
Collaborated with cross-functional teams to align data architecture with business and analytics needs.

Data Engineer

Arrow Dreams Technologies

Hyderabad , India

10.2011 - 02.2016

Designed and implemented scalable data pipelines to enhance data accessibility across departments.
Optimized ETL processes, ensuring accuracy and efficiency in data transformation workflows.
Collaborated with cross-functional teams to integrate new data sources into existing frameworks.
Led efforts to automate reporting systems, reducing manual tasks and improving turnaround time.

Education

Master of Science - Insurance And Risk Management

IIRM

Hyderabad

04-2011

B.Com Computers - Commerce, IT

Aurora Degree College

Hyderabad

04-2010

Skills

Data Lakehouse (Delta Lake), Medallion Architecture, Data Modeling, Distributed Systems, Data Mesh, Unity Catalog

Apache Spark (PySpark), Azure Databricks, Snowflake (Snowpipe, Tasks), Batch & Stream Processing

Python (Advanced), SQL (Expert – Performance Tuning), Java/Scala (Familiar)

Azure: ADLS Gen2, Azure Databricks
AWS: S3, EMR, Glue, Redshift (Working Knowledge)

GCP: BigQuery, Dataflow (Working Knowledge)
Snowflake, Delta Lake, Teradata, MySQL

Apache Kafka, Apache Airflow, Kubernetes (K8s), Docker, Terraform (IaC), CI/CD, Git

Data Governance, Data Lineage, Unity Catalog, Automated Data Quality & Reconciliation Frameworks, Power BI (DAX)