Chaitanya Gaddam
Denver,CO
Summary
Results-driven Big Data Engineer with 9+ years of experience in data analytics and product development, specializing in designing and optimizing large-scale data pipelines using Apache Spark (Scala) and Databricks. Demonstrated success delivering innovative, high-performance solutions across Hadoop and AWS ecosystems, consistently earning Outstanding Performance ratings and Quarterly Recognition awards. Skilled at leading end-to-end implementations, collaborating with cross-functional teams and clients, and translating complex business and technical requirements into scalable, actionable data solutions.
Overview
9
9
years of professional experience
Work History
Software Engineer III
Comcast
05.2021 - Current
- Designed and maintained scalable ETL pipelines using Apache Spark (Scala) on Databricks, processing over 5 TB of data weekly and reducing data pipeline run time by 65%.
- Developed and optimized Spark jobs in Scala on Databricks, improving data processing performance by 70% through advanced partitioning, caching, and efficient join strategies.
- Automated data metrics reporting using Databricks Workflows, reducing manual intervention by 80% and improving data accuracy.
- Led the migration of legacy Hadoop workloads to Databricks on AWS, reducing infrastructure costs by 30% and increasing system reliability.
- Built interactive Databricks notebooks (Scala and SQL) for data exploration, validation, and business reporting, enabling real-time insights for stakeholders.
- Provided technical leadership and mentorship, upskilling team members on Spark (Scala) and Databricks best practices.
- Designed and implemented a robust scheduling engine framework, resulting in a 40% reduction in Key Line Objectives (KLOs).
- Developed reusable frameworks across Hadoop and AWS platforms, enhancing scalability and reducing development time by 50%.
- Designed and implemented CI/CD pipelines with automated test cases, streamlining deployment cycles.
- Managed container-based deployments using Docker and Kubernetes, including image creation, registry management, and orchestration.
- Developed Tableau dashboards for workflow monitoring and data metric visualization, enabling data-driven decision-making.
Big Data Engineer (Contractor)
Comcast
08.2016 - 04.2021
- Migrated Hadoop applications to AWS and Databricks, improving scalability and reducing operational costs by 35%.
- Utilized Spark (Scala) and Hive for analytics and batch processing of large datasets, achieving a 60% reduction in data processing time.
- Developed Storm Trident topologies and established stable Kafka connection strategies for real-time data processing.
- Implemented GridGain cache processes to optimize lookup performance.
- Built cache structures on top of Accumulo and integrated with Storm for enhanced data processing efficiency.
- Processed large volumes of structured, semi-structured, and unstructured data using Spark (Scala) and Databricks.
- Developed a data ingestion framework using open-source Goblin code to consume data from Kafka and write to HDFS.
Education
M.S. - Computer Science
Wright State University
Dayton, OH04-2016
B.Tech. - Computer Science
JNTU
INDIA01.2013
Skills
- Apache Spark
- Programming Languages: Scala, Java, Python, C, C, SQL
- DevOps & Tools: CI/CD, Git, Docker Hub, Docker Registries
- Testing and debugging
- Advanced computer programming
- Efficient data management
- Solutions deployment
- Functional and integration testing
- Design and development
- Dashboard creation
ADDITIONAL INFORMATION
- Recognized with Outstanding Performance ratings and Quarterly Recognition awards.
- Strong communication and leadership skills; experienced in collaborating with clients and cross-functional teams.
- Proficient in PySpark and Spark SQL in addition to extensive Spark (Scala) experience.
Timeline
Software Engineer III
Comcast
05.2021 - Current
Big Data Engineer (Contractor)
Comcast
08.2016 - 04.2021
B.Tech. - Computer Science
JNTU
M.S. - Computer Science
Wright State University