Vamsidhar Challa - Senior Big Data Engineer - Dropbox, Inc

Summary

Results-driven Senior Big Data Engineer with 10+ years of experience designing scalable, cloud-native data architectures across AWS, GCP, and Azure. Expert in ETL/ELT development, data warehousing, distributed computing, and pipeline orchestration. Proven ability to lead data initiatives, mentor engineers, and collaborate with stakeholders to drive business decisions. Strong expertise in Snowflake, Databricks, Apache Spark, and Airflow. Passionate about performance optimization, cost efficiency, automation, and integrating AI/ML capabilities for data model enhancements and validation frameworks.

Overview

10

years of professional experience

5

years of post-secondary education

Work History

Senior Big Data Engineer

Dropbox, Inc

11.2021 - Current

Designed and optimized Snowflake, Databricks & Spark data models, improving data access efficiency and query speeds by 40%
Built and automated large-scale ETL/ELT pipelines, processing 1.5TB+ data per day using Snowflake, Databricks, Spark, Redshift, PSQL, and Fivetran
Led cross-functional collaboration with business, product, and engineering teams to strategize long-term data platform architecture
Developed role-based access control (RBAC) models in Snowflake, Databricks & AWS Lake Formation, ensuring compliance with GDPR and CCPA
Implemented cost tracking models in Snowflake, Databricks & Spark, improving cost visibility across teams by 30%
Designed complex Airflow DAGs to manage distributed data workflows, ensuring seamless automation and optimal execution
Managed stakeholder relationships across Finance, Data Science, Machine Learning, Business Intelligence, and HR teams, aligning data solutions with business objectives and compliance requirements
Established data governance policies and data contracts with upstream engineering teams, ensuring schema versioning, quality checks, and seamless data integration
Developed an AI-driven data validation framework, improving data integrity, automated anomaly detection, and feature engineering for ML models
Architected real-time streaming solutions using Kafka, Spark Streaming & AWS Kinesis, reducing data processing latency from minutes to real-time

Big Data Engineer

Maestro Technologies, Inc

09.2019 - 11.2021

Architected data pipelines for ETL/ELT processes in GCP (BigQuery, Dataflow, Cloud Functions)
Optimized Snowflake, Databricks & Spark warehouse models, reducing query execution time by 50%
Developed Kafka-based real-time streaming pipelines, reducing data ingestion delays from 10 minutes to
Managed data migrations from SQL Server, Oracle, and PostgreSQL to cloud-based BigQuery, Snowflake, Databricks & Spark environments
Integrated REST APIs with BigQuery, Snowflake, Databricks & Spark, enabling real-time data ingestion and analytics
Developed AI-powered data validation & anomaly detection models, improving data accuracy and operational monitoring

SQL Server Developer

Argus Information and Advisory Services, LLC

12.2015 - 09.2019

Developed SQL-based ETL processes, optimizing database performance through indexing, query tuning, and schema design
Designed and deployed SSIS packages for automated data integration and validation
Led data migration efforts, improving data retrieval speed and accuracy across OLTP and data mart environments

Education

Master’s in Electrical Engineering -

University of Missouri, Kansas City

01.2014 - 05.2015

Bachelor of Technology in Electrical & Electronics Engineering -

Koneru Lakshmaiah University, India

07.2009 - 06.2013

Skills

Cloud & Data Warehousing: Snowflake, Databricks, Redshift, BigQuery, Hive

ETL/ELT & Orchestration: Apache Airflow, Fivetran, Glue, Dataflow, ADF

Big Data Processing: Apache Spark, SparkSQL, PySpark, Kafka Streaming, Flink

Programming & Automation: Python (Pandas, NumPy, Scikit-learn), SQL, Shell Scripting

AI & Data Science Exposure: ML Model Integration, AI-driven Data Quality Validation, Feature Engineering

DevOps & CI/CD: Jenkins, Terraform, Azure DevOps, GitHub Actions, Kubernetes, Docker

Security & Governance: Snowflake RBAC, AWS Lake Formation, Data Compliance, GDPR/CCPA Standards

Reporting & Visualization: Tableau, Power BI, Looker

Accomplishments

✅ Optimized Snowflake, Databricks & Spark query performance, reducing cloud costs by 30% and improving execution speeds by 40%.
✅ Designed and implemented 60+ ETL/ELT pipelines, processing 1.5TB+ daily data across Snowflake, Databricks, Spark, and Redshift.
✅ Led migration of 750TB+ data from SQL Server to Snowflake, Databricks & Spark, reducing ETL execution time by 60%.
✅ Developed a homegrown ingestion tool for Snowflake, Databricks, Spark & BigQuery, reducing ingestion failures by 85%.
✅ Implemented RBAC & data governance policies in Snowflake, Databricks & AWS Lake Formation, ensuring compliance with industry standards.
✅ Designed and optimized data models for high-performance querying and storage efficiency, improving data retrieval speed by 50%and reducing storage costs.
✅ Established data contracts with upstream teams and stakeholders, ensuring consistent data quality, schema evolution, and versioning strategies.
✅ Integrated AI-powered validation frameworks into data pipelines, enhancing anomaly detection, schema validation, predictive analytics, and automated data integrity checks.
✅ Architected real-time streaming data pipelines using Kafka, Spark Streaming & AWS Kinesis, improving latency from minutes to sub-seconds.

Certifications Professional Development

Snowflake Certified Data Engineer
Microsoft Certified Azure Solution Developer
AWS Certified Data Analytics (In Progress)
Google Cloud Professional Data Engineer (In Progress)
Databricks Certified Data Engineer Associate

Technical Skills

✅ Leadership & Mentorship

✅ Stakeholder Collaboration

✅ Data Strategy & Architecture

✅ Analytical Problem-Solving

✅ Communication & Documentation

✅ AI & ML Data Strategy

Additional Information

Senior Data Engineer, Big Data Lead, Principal Data Engineer, Actively mentoring junior engineers in data engineering best practices., Passionate about optimizing data platforms for performance, security, and AI-driven insights.

Timeline