Vamsidhar Challa
Senior Big Data Engineer
San Francisco,CA
Summary
Results-driven Senior Big Data Engineer with 10+ years of experience designing scalable, cloud-native data architectures across AWS, GCP, and Azure. Expert in ETL/ELT development, data warehousing, distributed computing, and pipeline orchestration. Proven ability to lead data initiatives, mentor engineers, and collaborate with stakeholders to drive business decisions. Strong expertise in Snowflake, Databricks, Apache Spark, and Airflow. Passionate about performance optimization, cost efficiency, automation, and integrating AI/ML capabilities for data model enhancements and validation frameworks.
Overview
10
10
years of professional experience
5
5
years of post-secondary education
Work History
Senior Big Data Engineer
Dropbox, Inc
11.2021 - Current
- Designed and optimized Snowflake, Databricks & Spark data models, improving data access efficiency and query speeds by 40%
- Built and automated large-scale ETL/ELT pipelines, processing 1.5TB+ data per day using Snowflake, Databricks, Spark, Redshift, PSQL, and Fivetran
- Led cross-functional collaboration with business, product, and engineering teams to strategize long-term data platform architecture
- Developed role-based access control (RBAC) models in Snowflake, Databricks & AWS Lake Formation, ensuring compliance with GDPR and CCPA
- Implemented cost tracking models in Snowflake, Databricks & Spark, improving cost visibility across teams by 30%
- Designed complex Airflow DAGs to manage distributed data workflows, ensuring seamless automation and optimal execution
- Managed stakeholder relationships across Finance, Data Science, Machine Learning, Business Intelligence, and HR teams, aligning data solutions with business objectives and compliance requirements
- Established data governance policies and data contracts with upstream engineering teams, ensuring schema versioning, quality checks, and seamless data integration
- Developed an AI-driven data validation framework, improving data integrity, automated anomaly detection, and feature engineering for ML models
- Architected real-time streaming solutions using Kafka, Spark Streaming & AWS Kinesis, reducing data processing latency from minutes to real-time
Big Data Engineer
Maestro Technologies, Inc
09.2019 - 11.2021
- Architected data pipelines for ETL/ELT processes in GCP (BigQuery, Dataflow, Cloud Functions)
- Optimized Snowflake, Databricks & Spark warehouse models, reducing query execution time by 50%
- Developed Kafka-based real-time streaming pipelines, reducing data ingestion delays from 10 minutes to
- Managed data migrations from SQL Server, Oracle, and PostgreSQL to cloud-based BigQuery, Snowflake, Databricks & Spark environments
- Integrated REST APIs with BigQuery, Snowflake, Databricks & Spark, enabling real-time data ingestion and analytics
- Developed AI-powered data validation & anomaly detection models, improving data accuracy and operational monitoring
SQL Server Developer
Argus Information and Advisory Services, LLC
12.2015 - 09.2019
- Developed SQL-based ETL processes, optimizing database performance through indexing, query tuning, and schema design
- Designed and deployed SSIS packages for automated data integration and validation
- Led data migration efforts, improving data retrieval speed and accuracy across OLTP and data mart environments
Education
Master’s in Electrical Engineering -
University of Missouri, Kansas City
01.2014 - 05.2015
Bachelor of Technology in Electrical & Electronics Engineering -
Koneru Lakshmaiah University, India
07.2009 - 06.2013
Skills
Cloud & Data Warehousing: Snowflake, Databricks, Redshift, BigQuery, Hive
undefinedAccomplishments
- ✅ Optimized Snowflake, Databricks & Spark query performance, reducing cloud costs by 30% and improving execution speeds by 40%.
- ✅ Designed and implemented 60+ ETL/ELT pipelines, processing 1.5TB+ daily data across Snowflake, Databricks, Spark, and Redshift.
- ✅ Led migration of 750TB+ data from SQL Server to Snowflake, Databricks & Spark, reducing ETL execution time by 60%.
- ✅ Developed a homegrown ingestion tool for Snowflake, Databricks, Spark & BigQuery, reducing ingestion failures by 85%.
- ✅ Implemented RBAC & data governance policies in Snowflake, Databricks & AWS Lake Formation, ensuring compliance with industry standards.
- ✅ Designed and optimized data models for high-performance querying and storage efficiency, improving data retrieval speed by 50%and reducing storage costs.
- ✅ Established data contracts with upstream teams and stakeholders, ensuring consistent data quality, schema evolution, and versioning strategies.
- ✅ Integrated AI-powered validation frameworks into data pipelines, enhancing anomaly detection, schema validation, predictive analytics, and automated data integrity checks.
- ✅ Architected real-time streaming data pipelines using Kafka, Spark Streaming & AWS Kinesis, improving latency from minutes to sub-seconds.
Certifications Professional Development
- Snowflake Certified Data Engineer
- Microsoft Certified Azure Solution Developer
- AWS Certified Data Analytics (In Progress)
- Google Cloud Professional Data Engineer (In Progress)
- Databricks Certified Data Engineer Associate
Technical Skills
✅ Leadership & Mentorship
✅ Stakeholder Collaboration
✅ Data Strategy & Architecture
✅ Analytical Problem-Solving
✅ Communication & Documentation
✅ AI & ML Data Strategy
Additional Information
Senior Data Engineer, Big Data Lead, Principal Data Engineer, Actively mentoring junior engineers in data engineering best practices., Passionate about optimizing data platforms for performance, security, and AI-driven insights.
Timeline
Senior Big Data Engineer
Dropbox, Inc
11.2021 - Current
Big Data Engineer
Maestro Technologies, Inc
09.2019 - 11.2021
SQL Server Developer
Argus Information and Advisory Services, LLC
12.2015 - 09.2019
Master’s in Electrical Engineering -
University of Missouri, Kansas City
01.2014 - 05.2015
Bachelor of Technology in Electrical & Electronics Engineering -
Koneru Lakshmaiah University, India
07.2009 - 06.2013
Similar Profiles
Anna Lefkothea PapavasileiouAnna Lefkothea Papavasileiou
IT Support Specialist at DropboxIT Support Specialist at Dropbox
Ryan MaloneRyan Malone
Service Delivery Manager at DropboxService Delivery Manager at Dropbox
Anna Lefkothea PapavasileiouAnna Lefkothea Papavasileiou
IT Support Specialist at DropboxIT Support Specialist at Dropbox
Jonathan LatiefJonathan Latief
Field Research Intern at BIOMAField Research Intern at BIOMA
Andrea HernandezAndrea Hernandez
Hair Stylist/assistant at Acute CollectiveHair Stylist/assistant at Acute Collective