Summary
Overview
Work History
Education
Skills
Timeline
SoftwareEngineer

Kalyan Yalagandula

Clinton,MS

Summary

10+ years of experience in design, development, and implementation of robust technology systems, with specialized expertise in Hadoop Administration and Linux Administration. 6+ years of experience in Hadoop Administration & Big Data Technologies with cloud (Azure with insight & Google cloud and Aws) and 4 years of experience into Linux administration Experience with complete Software Design Lifecycle including design, development, testing and implementation of moderate to advanced complex systems. Hands on experience in installation, configuration, supporting and managing Hadoop Clusters using Horton works, Cloudera. Hadoop Cluster capacity planning, performance tuning, cluster Monitoring, Troubleshooting. Design Big Data solutions for traditional enterprise businesses. Excellent command in creating Backups & Recovery and Disaster recovery procedures and Implementing BACKUP and RECOVERY strategies for off-line and on-line Backups. Involved in bench marking Hadoop/ HBase cluster file systems various batch jobs and workloads. Making Hadoop cluster ready for development team working on POCs. Experience in minor and major upgrades of Hadoop and Hadoop eco system. Experience monitoring and troubleshooting issues with Linux memory, CPU, OS, storage, and network. Hands on experience in analyzing Log files for Hadoop and eco system services and finding root cause. Experience on Commissioning, Decommissioning, Balancing and Managing Nodes and tuning server for optimal performance of the cluster. As an admin involved in Cluster maintenance, trouble shooting, Monitoring, and followed proper backup & Recovery strategies. Good Experience in setting up the Linux environments, Password less SSH, creating file systems, disabling firewalls, Swappiness, Selinux and installing Java. Good Experience in Planning, Installing and Configuring Hadoop Cluster in Cloudera and Horton works Distributions. Installing and configuring Hadoop eco system like pig, hive. Hands on experience in Installing, Configuring, and managing the Hue and HCatalog. Experience in importing and exporting the data using Sqoop from HDFS to Relational Database systems/mainframe and vice-versa from enterprise data lake. Experience in importing and exporting the logs using Flume. Optimizing performance of HBase/Hive/Pig jobs. Hands on experience in Zookeeper and ZKFC in managing and configuring in Name Node failure scenarios. Handsome experience in Linux admin activities on RHEL & Cent OS. Experience in deploying Hadoop 2.0(YARN). Familiar with writing Oozie workflows and Job Controllers for job automation.

Overview

13
13
years of professional experience

Work History

Sr. Hadoop Administrator / Support Engineer

State Farm
04.2020 - Current
  • Currently working as admin on Cloudera (CDP 7.3.1) distribution for 12 clusters ranges from POC to PROD
  • Responsible for Cluster maintenance, Monitoring, commissioning, and decommissioning Data nodes, Troubleshooting, Manage and review data backups, Manage & review log files
  • Day to day responsibilities includes solving developer issues, deployments moving code from one environment to other environment, providing access to new users and providing instant solutions to reduce the impact and documenting the same and preventing future issues
  • Adding/installation of new components and removal of them through Cloudera Manager
  • Performance tuning on Hue and hive, impala, and spark jobs
  • Monitored workload, job performance and capacity planning using Cloudera Manager
  • Interacting with Cloudera support and log the issues in Cloudera portal and fixing them as per the recommendations
  • Cluster provisioning depend upon the user request and adding gateway nods to the cluster (CRMT tool)
  • Retrieved data from HDFS into relational databases with Sqoop
  • Parsed cleansed and mined useful and meaningful data in HDFS using Map-Reduce for further analysis
  • Implemented custom interceptors for flume to filter data and defined channel selectors to multiplex the data into different sinks
  • Partitioned and queried the data in Hive for further analysis by the BI team
  • Involved in extracting the data from various sources into Hadoop HDFS for processing
  • Worked on analyzing Hadoop cluster and different big data analytic tools including Pig, HBase database and Sqoop
  • Experienced in using Kafka as a distributed publisher-subscriber messaging system
  • Creating and truncating HBase tables in hue and taking backup of submitter ID(s)
  • Configuring, Managing permissions for the users in hue
  • Responsible for building scalable distributed data solutions using Hadoop
  • Commissioned and Decommissioned nodes on CDH5 Hadoop cluster on Red hat LINUX
  • Involved in loading data from LINUX file system to HDFS
  • Implemented test scripts to support test driven development and continuous integration
  • Worked on tuning the performance Pig queries
  • Experience in configuring the Storm in loading the data from MYSQL to HBASE using JMS
  • Responsible to manage data coming from different sources
  • Involved in loading data from UNIX file system to HDFS
  • Experience in managing and reviewing Hadoop log files
  • Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team
  • Installed Oozie workflow engine to run multiple Hive and pig jobs
  • Analyzed large amounts of data sets to determine optimal way to aggregate and report on it
  • Supported in setting up QA environment and updating configurations for implementing scripts with Pig and Sqoop
  • Environment: HDFS, Map reduce, Hive, Hue, Pig, Flume, Oozie, Sqoop, Cdh5, Apache Hadoop Spark, Solr, Storm, Cloudera Manager, Red hat, MySQL, and Oracle.

Hadoop admin & support engineer

Walmart labs INC
01.2019 - 03.2020
  • Experienced as a Hadoop admin & support engineer in Horton works (HDP 2.6.3) distribution clusters
  • Monitor Hadoop cluster connectivity and security
  • Manage and review Hadoop log files, File system management and monitoring
  • HDFS support and maintenance
  • Experienced in production support which involves solving the user incidents varies from P1 to P3 depend upon client service level agreement
  • Responsible for Cluster maintenance, Monitoring, commissioning, and decommissioning Data nodes, Troubleshooting, Manage & review log files
  • Working with data delivery teams to setup new Hadoop users
  • This job includes setting up Linux users, setting up Kerberos principals and testing HDFS, Hive, Spark and Map Reduce access for the new users
  • Involved in analyzing system failures, identifying root causes and recommend course of actions
  • Ensure that the Hadoop cluster is up and running all the time
  • Monitoring the cluster connectivity and performance
  • Injecting to Hadoop clusters from the enterprise data from data lakes
  • Manage and review Hadoop log files
  • Backup and recovery tasks
  • Supporting on Microsoft Azure HDInsight clusters
  • Deploy new clusters on Azure virtual machines by use cloud break
  • Providing the server support and maintenance Azure clusters
  • Installing and upgrading, configuring, and maintaining Hadoop eco components on Azure servers
  • Resource and security management
  • Troubleshooting application errors and ensuring that they do not occur again
  • Cluster maintenance as well as creation and removal of nodes using Hadoop Management Admin tools like Ambari
  • Collaborating with application teams to install operating system and Hadoop updates, patches, version upgrades when required
  • Providing ranger and Ranger kms access to HDFS and hive data bases
  • Experienced in using Kafka as a distributed publisher-subscriber messaging system
  • Import and export data back-end Tera data to Hadoop vice versa Hadoop to Tera data by use of Sqoop
  • Install and upgrade python packages on edge nodes
  • Monitoring the hive jobs and number on connections on hive met store through Grafana web ui
  • Hive performance tuning
  • Experienced on Ranger Centralized security administration to manage all security related tasks in a central UI or using REST APIs
  • Enhanced support for different authorization methods – By use of Ranger Role based access control
  • Centralize ranger auditing of user access and administrative actions (security related) within all the components of Hadoop
  • Environment: HDFS, Map reduce, yarn, zookeeper, Hive, Hue, Pig, Oozie, Sqoop, Apache Hadoop Spark

Sr. Software Engineer (Hadoop infrastructure analyst)

State Farm
01.2018 - 12.2018
  • Experienced as an admin on Cloudera (CDH 5.13.2) distribution for 12 clusters ranges from POC to PROD
  • And Horton works (2.5.3)
  • Responsible for Cluster maintenance, Monitoring, commissioning, and decommissioning Data nodes, Troubleshooting, Manage and review data backups, Manage & review log files
  • Day to day responsibilities includes solving developer issues, deployments moving code from one environment to other environment, providing access to new users and providing instant solutions to reduce the impact and documenting the same and preventing future issues
  • Adding/installation of new components and removal of them through Cloudera Manager
  • Collaborating with application teams to install operating system and Hadoop updates, patches, version upgrades
  • Worked on the journal node sync and Name node high availability and yarn, hue and oozie and Hbase and impala high availability
  • Experienced on Ranger and Atlas integration
  • And provide the atlas and ranger role-based policies
  • Performance tuning on Hue and hive, impala, and spark jobs
  • Monitored workload, job performance and capacity planning using Cloudera Manager
  • Interacting with Cloudera support and log the issues in Cloudera portal and fixing them as per the recommendations
  • Cluster provisioning depend upon the user request and adding gateway nods to the cluster (CRMT tool)
  • Retrieved data from HDFS into relational databases with Sqoop
  • Parsed cleansed and mined useful and meaningful data in HDFS using Map-Reduce for further analysis
  • Implemented custom interceptors for flume to filter data and defined channel selectors to multiplex the data into different sinks
  • Partitioned and queried the data in Hive for further analysis by the BI team
  • Involved in extracting the data from various sources into Hadoop HDFS for processing
  • Worked on analyzing Hadoop cluster and different big data analytic tools including Pig, HBase database and Sqoop
  • Sentry installation and configuration
  • Experienced in using kafka as a distributed publisher-subscriber messaging system
  • Creating and truncating HBase tables in hue and taking backup of submitter ID(s)
  • Configuring, Managing permissions for the users in hue
  • Responsible for building scalable distributed data solutions using Hadoop
  • Commissioned and Decommissioned nodes on CDH5 Hadoop cluster on Redhat LINUX
  • Involved in loading data from LINUX file system to HDFS
  • Implemented test scripts to support test driven development and continuous integration
  • Worked on tuning the performance Pig queries
  • Worked with application teams to install operating system, Hadoop updates, patches, version upgrades as required
  • Experience in configuring the Storm in loading the data from MYSQL to HBASE using JMS
  • Responsible to manage data coming from different sources
  • Involved in loading data from UNIX file system to HDFS
  • Experience in managing and reviewing Hadoop log files
  • Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team
  • Installed Oozie workflow engine to run multiple Hive and pig jobs
  • Analyzed large amounts of data sets to determine optimal way to aggregate and report on it
  • Supported in setting up QA environment and updating configurations for implementing scripts with Pig and Sqoop
  • Environment: HDFS, Map reduce, Hive, Hue, Pig, Flume, Oozie, Sqoop, Cdh5, Apache Hadoop Spark, Solr, Storm, Cloudera Manager, Red hat, MySQL, and Oracle.

Title Sr. Hadoop Operations & Administrator

All State
04.2017 - 12.2017
  • Experienced as admin in Horton works (HDP 2.5.1) distribution for 5 clusters ranges from POC to PROD
  • Cluster capacity planning depend upon the data usage
  • Designed and configure the Baston-Edge node configuration
  • Designed and configure HA of Hive & HBase services
  • Fine grained authorization to do a specific action and/or operation with Hadoop component/tool and managed through a central administration Ranger tool
  • Identify the root cause of zookeeper and spark logs (spark and zookeeper log are .out only) And collected all log files and integrated to cloud watch (AWS-ec2)
  • Changed the zookeeper and journal node edit directories (zookeeper and journal nodes has multiple directories)
  • Responsible for Cluster maintenance, Monitoring, commissioning, and decommissioning Data nodes, Troubleshooting, Manage and review data backups, Manage &review log files
  • Day to day responsibilities includes solving developer issues, deployments moving code from one environment to other environment, providing access to new users and providing instant solutions to reduce the impact and documenting the same and preventing future issues
  • Experienced on adding/installation of new components and removal of them through Ambari
  • Monitoring systems and services through Ambari dashboard to make the clusters available for the business
  • Architecture design and implementation of deployment, configuration management, backup, and disaster recovery systems and procedures
  • Hand on experience on cluster up gradation and patch upgrade without any data loss and with proper backup plans
  • Changing the configurations based on the requirements of the users for the better performance of the jobs
  • Experience using Devops tool Chef in provisioning AWS machines
  • Implemented a continuous deployment (CD) pipeline involving Jenkins, Chef to complete the automation from commit to deployment
  • Experienced on various components of Chef including chef server, workstations, Nodes, configurations of chef-client and writing cookbooks and recipes for Chef and assigning roles to nodes
  • Experienced in Ambari-alerts (critical & warning) configuration for various components and managing the alerts
  • Provided security and authentication with ranger where ranger admin provides administration and user sync adds the new users to the cluster
  • Good troubleshooting skills on Hue, which provides GUI for developer's/business users for day-to-day activities
  • Developed Map Reduce programs to cleanse the data in HDFS obtained from heterogeneous data sources to make it suitable for ingestion into Hive schema for analysis
  • Implemented complex Map Reduce programs to perform joins on the Map side using distributed cache
  • Implemented Name Node HA in all environments to provide high availability of clusters
  • Involved in snapshots and mirroring to maintain the backup of cluster data and even remotely
  • Experienced in managing and reviewing log files
  • (Identify the max backup index and max backup size of Log4j properties of all services in Hadoop)
  • Helping the users in production deployments throughout the process
  • Experienced in production support which involves solving the user incidents varies from sev1 to sev5
  • Managed and reviewed Log files as a part of administration for troubleshooting purposes
  • Communicate and escalate issues appropriately
  • As an admin followed standard Back up policies to make sure the high availability of cluster
  • Documented the systems processes and procedures for future references
  • Worked with systems engineering team to plan and deploy new environments and expand existing clusters
  • Monitored multiple clusters environments using AMBRI Alerts, Metrics
  • Environment: HDFS, Yarn, Map reduce, pig, zookeeper, spark, Kafka, Horton works, Ambari.

Software Engineer

Synchrony Financial
06.2016 - 03.2017
  • Experienced as admin in Horton works (HDP 2.2.4.2) distribution for 6 clusters ranges from POC to PROD
  • Responsible for Cluster maintenance, Monitoring, commissioning and decommissioning Data nodes, Troubleshooting, Manage and review data backups, Manage &review log files
  • Day to day responsibilities includes solving developer issues, deployments moving code from one environment to other environment, providing access to new users and providing instant solutions to reduce the impact and documenting the same and preventing future issues
  • Experienced on adding/installation of new components and removal of them through Ambari
  • Monitoring systems and services through Ambari dashboard to make the clusters available for the business
  • Architecture design and implementation of deployment, configuration management, backup, and disaster recovery systems and procedures
  • Hand on experience on cluster up gradation and patch upgrade without any data loss and with proper backup plans
  • Changing the configurations based on the requirements of the users for the better performance of the jobs
  • Experienced in Ambari-alerts configuration for various components and managing the alerts
  • Provided security and authentication with ranger where ranger admin provides administration and user sync adds the new users to the cluster
  • Experience on Standardize authorization Ranger Acees method across all Hadoop components
  • Good troubleshooting skills on Hue, which provides GUI for developer's/business users for day-to-day activities
  • Developed Map Reduce programs to cleanse the data in HDFS obtained from heterogeneous data sources to make it suitable for ingestion into Hive schema for analysis
  • Implemented Name Node HA in all environments to provide high availability of clusters
  • Capacity scheduler implementation in all environments to provide resources based on the allocation
  • Create queues and allocated the clusters resources to provide the priority for jobs
  • Experienced in Setting up the project and volume setups for the new projects
  • Involved in snapshots and mirroring to maintain the backup of cluster data and even remotely
  • Implementing the SFTP for the projects to transfer data from External servers to servers
  • Experienced in managing and reviewing log files
  • Working experience on maintaining MySQL databases creation and setting up the users and maintain the backup of cluster metadata databases with corn jobs
  • Setting up MySQL master and slave replications and helping business applications to maintain their data in MySQL Servers
  • Helping the users in production deployments throughout the process
  • Experienced in production support which involves solving the user incidents varies from sev1 to sev5
  • Managed and reviewed Log files as a part of administration for troubleshooting purposes
  • Communicate and escalate issues appropriately
  • As an admin followed standard Back up policies to make sure the high availability of cluster
  • Involved in Analyzing system failures, identifying root causes, and recommended course of actions
  • Documented the systems processes and procedures for future references
  • Worked with systems engineering team to plan and deploy new environments and expand existing clusters
  • Monitored multiple clusters environments using AMBRI Alerts, Metrics and Nagios
  • Environment: Hadoop HDFS, Map reduce, Hive, Pig, Flume, Oozie, Sqoop, Eclipse, Horton works, and Ambari.

Sr. Linux System Engineer

Cigniti Technologies
09.2012 - 04.2014
  • Installation and configuration of Linux for new build environment
  • Created Virtual server on Citrix Xen Server based host and installed operating system on Guest Servers
  • Configuring NFS, DNS
  • Updating YUM Repository and Red Hat Package Manager (RPM)
  • Created RPM packages using RPMBUILD, verifying the new build packages and distributing the package
  • Configuring distributed file systems and administering NFS server and NFS clients and editing auto-mounting mapping as per system / user requirements
  • Installation, configuration, and maintenance FTP servers, NFS, RPM and Samba
  • Configured SAMBA to get access of Linux shared resources from Windows
  • Created volume groups logical volumes and partitions on the Linux servers and mounted file systems on the created partitions
  • Deep understanding of monitoring and troubleshooting mission critical Linux machines
  • Experience with Linux internals, virtual machines, and open-source tools/platforms
  • Improve system performance by working with the development team to analyze, identify and resolve issues quickly
  • Ensured data recoverability by implementing system and application-level backups
  • Performed various configurations which include networking and Iptables, resolving hostnames, SSH keyless login
  • Managing Disk File Systems, Server Performance, Users Creation and Granting file access Permissions and RAID configurations
  • Support pre-production and production support teams in the analysis of critical services and assists with maintenance operations
  • Automate administration tasks through use of scripting and Job Scheduling using CRON
  • Environment: Linux, Citrix Xen Server 5.0, veritas volume manger and net back up.

Linux Administrator

People Link
01.2010 - 08.2012
  • Patched RHEL5 and Solaris 8, 9, 10 servers for EMC Power Path Upgrade for VMAX migration
  • Configured LVM (Logical Volume Manager) to manage volume group, logical and physical partitions and importing new physical volumes
  • Maintained and monitored all servers' operating system and application patch level, disk space and memory usage, user activities on daily basis, administration on Sun Solaris and RHEL systems, management archiving
  • Installed, configured, troubleshoot, and maintain Linux Servers and Apache Web server, configuration and maintenance of security and scheduling backups, submitting various types of corn jobs
  • Installations of HP Open view, monitoring tool, in more than 300 servers and worked with monitoring tools such as Nagios and HP Open view
  • Creation of VMs, cloning and migrations of the VMs on VMware vSphere 4.0/4.1
  • Setup and configured Apache to integrate with IBM WebSphere in load balancing environment
  • RHEL 4.1, Red hat Linux 8/9, IBM x series and HP ProLiant, Windows
  • Installing and upgrading OE & Red hat Linux and Solaris 8/9/10 x86 & SPARC on Servers like HP DL 380G3, 4 and 5 &Dell Power Edge servers
  • Environment: Linux 5/4, Sun Solaris 10/9/, Oracle 10g, SUN Servers, SUN Fires, Linux, HP open view service desk (OVSD), Kick start, Jump Start, Fujitsu Prime power servers, Samba.

Education

B. Tech - Computer Science

Kakatiya University

Master’s - software engineering

International Technological University

Skills

  • Technical Skills:
  • Hadoop ecosystem tool’s : Map Reduce, HDFS, Pig, Hive, Hbase, Sqoop, Zookeeper, Oozie, Hue, Storm, Kafka, Spark, Flume
  • Programming Language : Java, core java, HTML, Programming C, C, and PL/SQL
  • Operating Systems : Linux (RHEL, Ubuntu,), open Solaris, AIX
  • Scripting languages : Shell Scripting, HTML scripting,
  • Cloud Technologies : Aws, Google cloud platform, Microsoft Azure
  • Cluster Management Tools : HDP Ambari, Cloudera Manager, Hue

Timeline

Sr. Hadoop Administrator / Support Engineer

State Farm
04.2020 - Current

Hadoop admin & support engineer

Walmart labs INC
01.2019 - 03.2020

Sr. Software Engineer (Hadoop infrastructure analyst)

State Farm
01.2018 - 12.2018

Title Sr. Hadoop Operations & Administrator

All State
04.2017 - 12.2017

Software Engineer

Synchrony Financial
06.2016 - 03.2017

Sr. Linux System Engineer

Cigniti Technologies
09.2012 - 04.2014

Linux Administrator

People Link
01.2010 - 08.2012

B. Tech - Computer Science

Kakatiya University

Master’s - software engineering

International Technological University

Similar Profiles

CREATE PROFILE
Kalyan Yalagandula