Thangabalaji Swamynathan
Senior Data Engineer
Tampa,FL
Summary
With over 12 years of experience as a seasoned IT professional, specialization lies in distributed systems within Hybrid Cloud Technologies. Specialized in crafting Enterprise Data lake/Lake house using the AWS/Azure Cloud technology stack, consistently delivering high-quality solutions.
Proficient in multitasking, interactive, and leadership skills, thriving in dynamic environments, meeting challenging deadlines, and maintaining service quality. Skill set covers a spectrum of Azure/AWS Cloud Technologies, including Apache Spark, Airflow, Confluent Kafka, Snowflake, Kubernetes, the Hadoop Ecosystem, and various integration technologies vital to ETL pipelines.
A notable career achievement involves orchestrating seamless migration of topics and applications from managed Kafka in Kubernetes (K8s) to the Confluent Cloud platform. Another highlight encompasses architecture and construction of a Hybrid AWS Cloud-based Big Data Pipeline, resulting in a remarkable 64% reduction in processing time and annual cost savings of $750,000.
Enthusiastic about leveraging refined skills and extensive experience to contribute to team and organizational success.
Overview
12
12
years of professional experience
4
4
years of post-secondary education
2
2
Certifications
Work History
Senior Data Engineer
Footlocker
St. Petersburg, FL
04.2021 - Current
- Displayed sound understanding of standard networking protocols, virtual networks and load balancing.
- Defined cloud architecture for both hybrid and non-hybrid cloud solutions.
- Identified and remediated single point failure and security risks.
- Developed and documented system security authorization boundaries for cloud-based solutions and client applications within cloud service.
- Designed and implemented customer identity graphs using Confluent Kafka, Spark streaming, Dgraph, and Spring Boot applications, enhancing personalization of marketing campaigns and operational intelligence in Azure cloud.
- Migrated legacy reporting environment to Snowflake and PowerBI, reducing data processing time by 40% and improving data visualization for business stakeholders.
- Led adoption of modern data platforms (Apache Spark, Airflow, Confluent Kafka, Snowflake, Kubernetes, and AKS) for 30% increase in data processing efficiency.
- Successfully led migration of topics and applications from managed Kafka running in Kubernetes (K8s) to Confluent Cloud (PaaS).
- Implemented CI/CD pipelines and infrastructure as code templates using terraform, reducing deployment time by 50% and improving development efficiency.
- Developed and maintained infrastructure as code to ensure scalability and reliability.
- Developed and deployed Kafka connect applications for Mongodb, Sql Server and snowflake.
- Proficient in creating various types of documentation, including functional, process, and test plans.
- Demonstrated ability to deliver clear and concise documentation that effectively communicates complex concepts.
- Collaborated with cross-functional teams to ensure seamless integration and compatibility.
Data Platform Engineer
Footlocker
St. Petersburg, FL
08.2019 - 03.2021
- Guided and influenced existing partners on recommended upgrades and enhancements to integrated solutions.
- Conducted research to evaluate systems design and process efficiency.
- Designed and implemented scalable Big Data applications using Spark, Kafka, and AWS technologies, improving data processing speed by 30%.
- Built real-time pipelines for data ingestion from various sources using Spark Streaming and Kafka Connect, enabling near real-time analytics.
- Developed RESTful web services with Spring Boot to expose data, improving data accessibility and integration capabilities.
- Optimized and performance-tuned Spark jobs and Hive queries, reducing data processing time by 25%.
- Followed Agile and Scrum principles in project development, ensuring timely delivery and effective collaboration.
Big Data Engineer
Nielsen
Oldsmar, FL
06.2017 - 08.2019
- Orchestrated and built Big Data Pipeline project in separate Hybrid Cloud (AWS) Account for Digital AdRatings Project, achieving reduction in Processing Time from 14 hrs. to 5 hrs. and realizing Operating cost savings of $750K/Year.
- Contributed to Batch Processing One Billion Records as Daily ETL Workload.
Established Automated CI/CD setup and deployment of code using Bitbucket, Jenkins, and Sonar Qube. - Implemented Test Driven Development with test data set up in local environment (Local PC) for cost-effective cluster-free development.
- Developed code in Spark & Scala Functional Approach with Automated Test cases for each feature/Module, achieving 100% local development coverage.
- Utilized Hadoop cluster (EMR) for Compute, storing KMS Encrypted Data and code in S3 Storage to run spot clusters, reducing cost by 80%.
- Leveraged Spot Instances and homegrown solution for cost-efficient multi-Availability Zone switching during high demand.
- Set up Elastic Search Logstash Kibana (ELK) stack, feeding real-time logs using AWS Kinesis from Hadoop Cluster (EMR) for Log Analytics.
Employing Next-Gen Orchestration Technology - Airflow for job scheduling and monitoring. - Provisioned Infrastructure as code using Terraform Software.
- Enabled Data Science with Read-only Access to Data Bricks clusters for processing data in DAR S3 Account, facilitating Data science Analytics and Insights.
- Implemented Dockerized solutions, ECS, AWS Lambda for Event triggers, Athena for Serverless querying, and more.
Data Engineer /Data Warehousing Specialist
Nielsen
Chennai
01.2015 - 12.2016
- Demonstrated Experience in Building Enterprise Data Lake in AWS Cloud.
Designing and Developing ETL framework in Hadoop using Sqoop, Oozie, Distcp, HDFS. - Building components to extract and load data from any RDBMS source like Netezza, Redshift, Oracle, Mysql to Hadoop File system.
- Building components to extract and load from Hadoop file system, S3 Storage using Distcp and Hive with MR and Tez engine.
- Designing and Building Materialistic views in Hive as S3 data storage and metadata in Metastore.
- Building components to load incremental data into materialistic views whenever there is change in base objects.
- Building components to replicate and sync Metastore between two or more clusters.
- Storing and retrieving data in Encrypted S3 and RDS in AWS.
- Building data validation and comparison module between any RDBMS systems vs. HDFS using Apache Spark.
- Applying business logic rules in-memory processing using AWS EMR Spark.
- Working with multiple security data layer like Kerberos for Authentication for data stored in cluster and Sentry for Authorization of data stored in HDFS or S3.
- Migrating ETL framework build in Hadoop Cloudera to AWS Elastic Map Reduce Service.
- Running AWS EMR in Spot instances and scaling up/down core nodes based on demand.
- Building Transient EMR cluster using AWS lambda and AWS Cloud Formation Stack.
- Developed applications and designed processes for transformation and data management from company-wide databases.
- Generated reports, maintaining dimensional as well as relational data structures and managing operational data store and data warehouse.
Software Engineer /Technical Lead Engineer
Nielsen
Chennai
01.2013 - 05.2015
- Reviewed project specifications and designed technology solutions that met or exceeded performance expectations.
- Coordinated with other engineers to evaluate and improve software and hardware interfaces.
- Worked with software development and testing team members to design and develop robust solutions to meet client requirements for functionality, scalability, and performance.
- Identified and documented project changes with proactive budget oversight.
- Develop the new programs based on the client request.
Providing Production support for the PRO applications and coordinating the development projects. - Developed a Application to monitor and report anomaly for the PC/Mobile TV rating Homes for Nielsen Media Research.
- Reported daily status to Technical Delivery Managers and Management.
- Responsible for the evaluation of design and development of the project lifecycle.
- Responsible for coding, component integration testing and unit testing.
- Created and updated system's functional and reference documentation.
- Worked on Production fixes.
- Created test cases and recorded test results for new/enhanced requests.
- Setup the Environments for QA’s and coordinates in testing.
- Verifying the QA Test results and quickly fixing the logs if any.
Data Engineer
Nielsen
Chennai
02.2012 - 04.2013
- Collaborated on stages of systems development lifecycle from requirement gathering to production releases.
- Collaborated with project managers to select ambitious, but realistic coding milestones on pre-release software project development.
- Revised, modularized and updated old code bases to modern development standards, reducing operating costs, and improving functionality.
- Documented technical workflows and knowledge to educate newly hired employees.
- Analysis of requirements.
- Designing of program based on the client requirement.
- Preparation of impact analysis and design document.
- Employed vast knowledge of relational database concepts and PL/SQL coding to produce highly efficient data systems.
- Document the test results and prepare the UTR.
- Providing Production support for the application.
- Prepare and Analyze the design which helped throughout the software development life cycle.
- Implementation of developed/restructured components in to QA region.
- Co-ordinate with Release Management Team.
- Documentation of Programs and Job Flows.
Education
Bachelor in Engineering - Electrical And Electronics Engineering
Anna University
K S Rangasamy College Of Technology 08.2007 - 05.2011
Skills
AWS Cloud Services – S3, EC2, RDS, EMR, DMS, SQS, SNS, EKS, ELB, IAM, VPC, CloudWatch, Kinesis, CloudFormation
AWS Data Ingestion Services – DynamoDB, Redshift, Athena, Step Functions, Glue, Lambda, Lake Formation
AI / ML Skills – AWS SageMaker, PyTorch, Spark ML
Programming Skills – Unix, Python, Scala, SQL
Cloud Datawarehouse – Snowflake Data Cloud, SnowSQL, Snowsight, Snowpipe
Data Visualization / BI Tools – Microsoft PowerBI
Data Modeling Tools – PowerBI Designer, Erwin
Big Data – Cloudera Hadoop, Hive, Apache Spark, Sqoop, Kafka, Impala
Containerization Services – Kubernetes,AKS,ECS
CI / CD Tools – Jenkins, Urban CodeDeploy, SVN, Git, Terraform,
Scheduling/Orchestration Tools – Control M, Oozie,, Apache Airflow
Data Monitoring Tools – Kibana, New Relic, Grafana,Promethes,CloudWatch
ETL / ELT Tools – Talend (Open Studio, Data Integration Suite), dbt (Data Build Tool)
Stream Processing -Confluent Kafka, Spark Streaming,Azure service bus
Cloud Architecture Design
Certification
CCA175: CCA Spark and Hadoop Developer
Timeline
Senior Data Engineer
Footlocker
04.2021 - Current
Data Platform Engineer
Footlocker
08.2019 - 03.2021
Big Data Engineer
Nielsen
06.2017 - 08.2019
CCA175: CCA Spark and Hadoop Developer
06-2017
Data Engineer /Data Warehousing Specialist
Nielsen
01.2015 - 12.2016
ITIL® 2011 Foundation
12-2013
Software Engineer /Technical Lead Engineer
Nielsen
01.2013 - 05.2015
Data Engineer
Nielsen
02.2012 - 04.2013
Bachelor in Engineering - Electrical And Electronics Engineering
Anna University
08.2007 - 05.2011