Summary
Overview
Work History
Education
Skills
Certification
Timeline
Generic

Thangabalaji Swamynathan

Senior Data Engineer
Tampa,FL

Summary

With over 12 years of experience as a seasoned IT professional, specialization lies in distributed systems within Hybrid Cloud Technologies. Specialized in crafting Enterprise Data lake/Lake house using the AWS/Azure Cloud technology stack, consistently delivering high-quality solutions.

Proficient in multitasking, interactive, and leadership skills, thriving in dynamic environments, meeting challenging deadlines, and maintaining service quality. Skill set covers a spectrum of Azure/AWS Cloud Technologies, including Apache Spark, Airflow, Confluent Kafka, Snowflake, Kubernetes, the Hadoop Ecosystem, and various integration technologies vital to ETL pipelines.

A notable career achievement involves orchestrating seamless migration of topics and applications from managed Kafka in Kubernetes (K8s) to the Confluent Cloud platform. Another highlight encompasses architecture and construction of a Hybrid AWS Cloud-based Big Data Pipeline, resulting in a remarkable 64% reduction in processing time and annual cost savings of $750,000.

Enthusiastic about leveraging refined skills and extensive experience to contribute to team and organizational success.

Overview

12
12
years of professional experience
4
4
years of post-secondary education
2
2
Certifications

Work History

Senior Data Engineer

Footlocker
St. Petersburg, FL
04.2021 - Current
  • Displayed sound understanding of standard networking protocols, virtual networks and load balancing.
  • Defined cloud architecture for both hybrid and non-hybrid cloud solutions.
  • Identified and remediated single point failure and security risks.
  • Developed and documented system security authorization boundaries for cloud-based solutions and client applications within cloud service.
  • Designed and implemented customer identity graphs using Confluent Kafka, Spark streaming, Dgraph, and Spring Boot applications, enhancing personalization of marketing campaigns and operational intelligence in Azure cloud.
  • Migrated legacy reporting environment to Snowflake and PowerBI, reducing data processing time by 40% and improving data visualization for business stakeholders.
  • Led adoption of modern data platforms (Apache Spark, Airflow, Confluent Kafka, Snowflake, Kubernetes, and AKS) for 30% increase in data processing efficiency.
  • Successfully led migration of topics and applications from managed Kafka running in Kubernetes (K8s) to Confluent Cloud (PaaS).
  • Implemented CI/CD pipelines and infrastructure as code templates using terraform, reducing deployment time by 50% and improving development efficiency.
  • Developed and maintained infrastructure as code to ensure scalability and reliability.
  • Developed and deployed Kafka connect applications for Mongodb, Sql Server and snowflake.
  • Proficient in creating various types of documentation, including functional, process, and test plans.
  • Demonstrated ability to deliver clear and concise documentation that effectively communicates complex concepts.
  • Collaborated with cross-functional teams to ensure seamless integration and compatibility.

Data Platform Engineer

Footlocker
St. Petersburg, FL
08.2019 - 03.2021
  • Guided and influenced existing partners on recommended upgrades and enhancements to integrated solutions.
  • Conducted research to evaluate systems design and process efficiency.
  • Designed and implemented scalable Big Data applications using Spark, Kafka, and AWS technologies, improving data processing speed by 30%.
  • Built real-time pipelines for data ingestion from various sources using Spark Streaming and Kafka Connect, enabling near real-time analytics.
  • Developed RESTful web services with Spring Boot to expose data, improving data accessibility and integration capabilities.
  • Optimized and performance-tuned Spark jobs and Hive queries, reducing data processing time by 25%.
  • Followed Agile and Scrum principles in project development, ensuring timely delivery and effective collaboration.

Big Data Engineer

Nielsen
Oldsmar, FL
06.2017 - 08.2019
  • Orchestrated and built Big Data Pipeline project in separate Hybrid Cloud (AWS) Account for Digital AdRatings Project, achieving reduction in Processing Time from 14 hrs. to 5 hrs. and realizing Operating cost savings of $750K/Year.
  • Contributed to Batch Processing One Billion Records as Daily ETL Workload.
    Established Automated CI/CD setup and deployment of code using Bitbucket, Jenkins, and Sonar Qube.
  • Implemented Test Driven Development with test data set up in local environment (Local PC) for cost-effective cluster-free development.
  • Developed code in Spark & Scala Functional Approach with Automated Test cases for each feature/Module, achieving 100% local development coverage.
  • Utilized Hadoop cluster (EMR) for Compute, storing KMS Encrypted Data and code in S3 Storage to run spot clusters, reducing cost by 80%.
  • Leveraged Spot Instances and homegrown solution for cost-efficient multi-Availability Zone switching during high demand.
  • Set up Elastic Search Logstash Kibana (ELK) stack, feeding real-time logs using AWS Kinesis from Hadoop Cluster (EMR) for Log Analytics.
    Employing Next-Gen Orchestration Technology - Airflow for job scheduling and monitoring.
  • Provisioned Infrastructure as code using Terraform Software.
  • Enabled Data Science with Read-only Access to Data Bricks clusters for processing data in DAR S3 Account, facilitating Data science Analytics and Insights.
  • Implemented Dockerized solutions, ECS, AWS Lambda for Event triggers, Athena for Serverless querying, and more.

Data Engineer /Data Warehousing Specialist

Nielsen
Chennai
01.2015 - 12.2016
  • Demonstrated Experience in Building Enterprise Data Lake in AWS Cloud.
    Designing and Developing ETL framework in Hadoop using Sqoop, Oozie, Distcp, HDFS.
  • Building components to extract and load data from any RDBMS source like Netezza, Redshift, Oracle, Mysql to Hadoop File system.
  • Building components to extract and load from Hadoop file system, S3 Storage using Distcp and Hive with MR and Tez engine.
  • Designing and Building Materialistic views in Hive as S3 data storage and metadata in Metastore.
  • Building components to load incremental data into materialistic views whenever there is change in base objects.
  • Building components to replicate and sync Metastore between two or more clusters.
  • Storing and retrieving data in Encrypted S3 and RDS in AWS.
  • Building data validation and comparison module between any RDBMS systems vs. HDFS using Apache Spark.
  • Applying business logic rules in-memory processing using AWS EMR Spark.
  • Working with multiple security data layer like Kerberos for Authentication for data stored in cluster and Sentry for Authorization of data stored in HDFS or S3.
  • Migrating ETL framework build in Hadoop Cloudera to AWS Elastic Map Reduce Service.
  • Running AWS EMR in Spot instances and scaling up/down core nodes based on demand.
  • Building Transient EMR cluster using AWS lambda and AWS Cloud Formation Stack.
  • Developed applications and designed processes for transformation and data management from company-wide databases.
  • Generated reports, maintaining dimensional as well as relational data structures and managing operational data store and data warehouse.

Software Engineer /Technical Lead Engineer

Nielsen
Chennai
01.2013 - 05.2015
  • Reviewed project specifications and designed technology solutions that met or exceeded performance expectations.
  • Coordinated with other engineers to evaluate and improve software and hardware interfaces.
  • Worked with software development and testing team members to design and develop robust solutions to meet client requirements for functionality, scalability, and performance.
  • Identified and documented project changes with proactive budget oversight.
  • Develop the new programs based on the client request.
    Providing Production support for the PRO applications and coordinating the development projects.
  • Developed a Application to monitor and report anomaly for the PC/Mobile TV rating Homes for Nielsen Media Research.
  • Reported daily status to Technical Delivery Managers and Management.
  • Responsible for the evaluation of design and development of the project lifecycle.
  • Responsible for coding, component integration testing and unit testing.
  • Created and updated system's functional and reference documentation.
  • Worked on Production fixes.
  • Created test cases and recorded test results for new/enhanced requests.
  • Setup the Environments for QA’s and coordinates in testing.
  • Verifying the QA Test results and quickly fixing the logs if any.

Data Engineer

Nielsen
Chennai
02.2012 - 04.2013
  • Collaborated on stages of systems development lifecycle from requirement gathering to production releases.
  • Collaborated with project managers to select ambitious, but realistic coding milestones on pre-release software project development.
  • Revised, modularized and updated old code bases to modern development standards, reducing operating costs, and improving functionality.
  • Documented technical workflows and knowledge to educate newly hired employees.
  • Analysis of requirements.
  • Designing of program based on the client requirement.
  • Preparation of impact analysis and design document.
  • Employed vast knowledge of relational database concepts and PL/SQL coding to produce highly efficient data systems.
  • Document the test results and prepare the UTR.
  • Providing Production support for the application.
  • Prepare and Analyze the design which helped throughout the software development life cycle.
  • Implementation of developed/restructured components in to QA region.
  • Co-ordinate with Release Management Team.
  • Documentation of Programs and Job Flows.

Education

Bachelor in Engineering - Electrical And Electronics Engineering

Anna University
K S Rangasamy College Of Technology
08.2007 - 05.2011

Skills

    AWS Cloud Services – S3, EC2, RDS, EMR, DMS, SQS, SNS, EKS, ELB, IAM, VPC, CloudWatch, Kinesis, CloudFormation

undefined

Certification

CCA175: CCA Spark and Hadoop Developer

Timeline

Senior Data Engineer

Footlocker
04.2021 - Current

Data Platform Engineer

Footlocker
08.2019 - 03.2021

Big Data Engineer

Nielsen
06.2017 - 08.2019

CCA175: CCA Spark and Hadoop Developer

06-2017

Data Engineer /Data Warehousing Specialist

Nielsen
01.2015 - 12.2016

ITIL® 2011 Foundation

12-2013

Software Engineer /Technical Lead Engineer

Nielsen
01.2013 - 05.2015

Data Engineer

Nielsen
02.2012 - 04.2013

Bachelor in Engineering - Electrical And Electronics Engineering

Anna University
08.2007 - 05.2011
Thangabalaji SwamynathanSenior Data Engineer