Bhavya Mallela

• Led the migration of 50+ large-scale Scala Spark ETL jobs to modular PySpark pipelines, improving code maintainability, reducing runtime, and aligning all workflows with reusable components and clean architecture standards.
• Automated and scheduled migrated ETL pipelines using Rundeck and Concourse, implementing multi-stage environments, versioned deployments, and automated validation steps to ensure stable production releases.
• Re-engineered a legacy CouchDB-based ETL by rewriting it in Python/PySpark to support MongoDB as the new scalable NoSQL backend; enabled region-wise S3 ingestion, transformation, and bulk write operations to Mongo for downstream client application usage.
• Optimized MongoDB update ETL by implementing checkpointing, micro-batch processing, broadcast joins, selective partitioning, incremental comparison logic, and Adaptive Query Execution (AQE) / adaptive shuffling, reducing overall runtime and preventing stage failures during multi-million–record (15Million) updates.
• Designed and supported Ingestion, Compute, and Advanced Compute layers that extract large datasets from Teradata, Oracle, Trino/Presto, MinIO, PostgreSQL, and SQL Server, apply business rule transformations, unify schemas, and publish region-wise JSON outputs into AWS S3.
• Implemented complex PySpark transformations including data cleansing, schema alignment, JSON flattening/unflattening, CDC logic, multi-source merging, and Delta/Parquet optimization techniques for high-volume telecom datasets.
• Designed and implemented AWS-based data pipelines using Amazon S3 for data lake storage, AWS Glue/Athena for metadata and querying, and IAM for secure, role-based access across environments.
• Built serverless and event-driven workflows on AWS leveraging Lambda, API Gateway, and SQS/SNS to orchestrate ingestion, trigger ETL jobs, and handle asynchronous processing at scale.
• Monitored and optimized production workloads on AWS using CloudWatch metrics, logs, and alarms, performing performance tuning, failure analysis, and cost optimization for high-volume batch and micro-batch processing jobs.
• Integrated AI-powered capabilities into internal engineering platforms, enabling teams to apply AI-driven insights for code quality analysis, impact assessment, and faster development cycles across multiple projects.
• Worked with AI/LLM-based services to enhance developer workflows, supporting use cases such as intelligent code suggestions, automated review assistance, and contextual documentation generation within enterprise repositories.
• Partnered with cross-functional teams to operationalize AI solutions at scale, focusing on secure access, governance, performance monitoring, and seamless adoption of AI features within existing cloud and data engineering ecosystems.
• Improved pipeline performance through advanced Spark optimization techniques including caching strategies, predicate/partition pruning, Z-Ordering (if applicable), file compaction, and minimizing skewed joins using salting and repartitioning.
• Built and enhanced business-critical eligibility and device program pipelines involving complex conditional rule mapping, window functions, and join strategies, ensuring accurate downstream data for multiple service teams.
• Collaborated closely with business stakeholders to understand data requirements, perform quick SQL/PySpark analysis, deliver ad hoc datasets, and validate rule logic across divisions and regions.
• Worked cross-functionally with product owners, analysts, QA, and DevOps teams to ensure accurate data delivery, consistent releases, code reviews, and seamless integration between ingestion, compute, and application-facing layers.

Performed batch and micro-batch data processing using PySpark, transforming structured and semi-structured datasets across telecom and healthcare domains.

• Designed and executed end-to-end ETL pipelines on Azure, using Azure Data Factory (ADF) for orchestration, Azure Data Lake / Blob Storage for scalable storage, and Azure SQL Database / Azure Synapse Analytics for analytical workloads.

• Built and optimized distributed data processing pipelines on Azure Databricks, leveraging PySpark and Spark SQL for large-scale transformations, performance tuning, and curated data delivery.

• Worked extensively with enterprise databases and data warehouses including Teradata, PostgreSQL, SQL Server, Oracle, and Snowflake, supporting both analytical and transactional workloads.

• Designed and optimized complex SQL queries (joins, CTEs, window functions, aggregations) to support ETL pipelines, reporting, and downstream analytics with high data accuracy.

• Led data migration and modernization initiatives, migrating data from Teradata and Oracle to cloud-based analytics platforms, ensuring schema alignment, data reconciliation, and query parity.

• Applied database performance tuning techniques such as indexing, partitioning, statistics management, and query plan analysis to improve execution times and resource efficiency.

• Built serverless workflows on AWS using Lambda, S3 triggers, and CloudWatch Events to automate ingestion and post-processing, improving reliability and reducing operational overhead.

• Implemented secure cloud access and governance using AWS IAM, Secrets Manager, encryption, and Azure RBAC, ensuring compliance across cloud ETL pipelines.

• Improved analytical query performance in Azure Synapse Analytics using columnstore indexing, CTAS patterns, and workload optimization.

• Supported production cloud workloads with strong focus on monitoring, reliability, and incident resolution, using Azure Monitor and AWS CloudWatch.

• Worked in Agile/Scrum environments, collaborating with cross-functional teams and leveraging Git, GitHub, CI/CD practices, Jira, and Confluence for delivery and documentation.

• Wrote and optimized SQL queries to extract, join, and filter data from relational databases such as MySQL, PostgreSQL, and SQL Server, supporting business reports and ad hoc data requests.
• Developed interactive dashboards and visualizations using Tableau, including bar charts, filters, KPIs, and calculated fields, helping stakeholders derive insights from structured data.
• Built simple web-based dashboards using HTML, CSS, and JavaScript, integrating backend data through APIs or static datasets for internal reporting interfaces.
• Gained hands-on exposure to AWS services like S3 for data storage, RDS for managing relational databases, and Athena for querying S3 data using SQL.
• Supported basic ETL workflows by assisting with data validation, cleaning, and transformation tasks under guidance, learning industry best practices for scalable and reusable data pipelines.
• Collaborated with cross-functional teams to define project requirements and deliver software solutions.