Madhu
Irving,TX
Summary
- Excellent working experience in DevOps/DevSecOps process and tools area (Code review, Unit test automation, Build & Release automation, Security Operations, Service, Incident and Change Management).
- Built tools and services (in Python, Ruby) for supporting operational excellence, tracking/improving infrastructure efficiency, infrastructure/operations automation and built consumable data streams for log analytics and BI.
- Provisioned and configured AWS EC2 instances for SIT & QA environments on demand and experienced with spin-off instances via AWS Management console, Jenkins EC2 Plugin and Cloud Formation Templates.
- Extensive knowledge and Experience in Deploying applications in cloud servers like AWS and implementation knowledge on Azure clouds.
- Good experience in automating build process using Python, Groovy, Shell. Good experience in automating build process using Python, Groovy, Shell.
- Experienced in Azure IaaS - Virtual Networks, Virtual Machines, Cloud Services, Resource Groups, Express Route, Traffic Manager, VPN, Load Balancing, Application Gateways, and Auto-Scaling.
- Strong expertise on DevOps concepts like Continuous Integration (CI), Continuous delivery (CD) and Infrastructure as Code, Cloud Computing etc.
- Hand on experience with Docker administration and build Docker images by writing Docker files for various applications deployments.
- Building, Installation, Administration, System Monitoring, Performance Tuning, Troubleshooting, Maintenance of Linux operating systems.
- Experience in branching, tagging, and maintaining the Version Control and Source Code management tools like GIT, SVN (subversion) on Linux and windows platforms.
- Good Knowledge on AWS console administrations as well as CLI operations.
- Design AWS Cloud Formation templates to create custom sized VPC, subnets, NAT to ensure successful deployment of Web applications and database templates.
- Built Automation and Build Pipeline Development using Jenkins and Maven. Set up build pipelines in Jenkins by using various plugins like Maven plugin, EC2 plugin etc.
- Expertise in managing Amazon instances by taking AMIs and performing administration and monitoring of Amazon instances using Amazon Cloud Watch.
- Experience in Managing deployment, maintenance, support and upgrade of servers, hardware, software, operating systems in Server Environment.
- Expertise in using Linux servers and Configuration Management tools like Puppet, Chef, Ansible.
- Experience in writing Ansible Playbooks for installing operating systems, installing Software, web applications in Virtual Machine and AWS EC2instances.
- Excellent understanding of Network topologies such as LAN, WAN, VPN and Firewalls.
- Experience is using Tomcat, JBOSS, Web logic and Web Sphere application servers for deployment.
- Experienced in Amazon AWS Cloud infrastructure services like EC2, VPC, S3, SNS, SQS, IAM, RDS, SES, Route53, Cloud Watch, Cloud Front, Elastic Load Balancers, and Cloud Trial.
- Setting up data in AWS using the S3 bucket and configuring instance backups to the S3 bucket.
- Recreating existing application logic and functionality in the Azure Data Lake, Data Factory, SQL Database, and SQL data warehouse environment.
- Responsible for building scalable distributed data solutions in both batch and streaming mode on Google Big Query using Kafka, Spark, and Core Java.
- Evaluate Snowflake Design considerations for any change in the application.
- Build the Logical and Physical data model for Snowflake as per the changes required.
- Define virtual warehouse sizing for Snowflake for different types of workloads.
- Developed Talend ETL jobs to push the data into Talend MDM and develop the jobs to extract the data from MDM.
- Designed dimensional model, data lake architecture, and data vault 2.0 on Snowflake and used Snowflake logical data warehouse for computing.
- Well-versed with Snowflake features like clustering, time travel, cloning, logical data warehouse, caching, etc.
- Exceptional skills in SQL server reporting services, analysis services, Tableau, and data visualization tools.
- Worked on Oracle Databases, RedShift, and Snowflakes
- Define virtual warehouse sizing for Snowflake for different types of workloads.
- Exceptionally coordinated with the ability to deal with numerous projects and fulfill time constraints and can work cooperatively with all the colleagues to guarantee quality.
Overview
10
10
years of professional experience
1
1
Certification
Work History
Data Engineer
AT&T
07.2023 - Current
- Worked extensively with Microsoft Azure Cloud services such as Application Gateways, Load Balancing, Azure Active Directory(AD), Azure Resource Manager(ARM), Blob Storage and SQL Database.
- Assisted in setting up end-to-end pipelines in Azure DevOps using YAML scripts and configuring service connections in across multiple Projects in Azure DevOps Organization.
- Utilized Azure DevOps Services to create Azure Pipelines for Build and Release and Azure Boards for planning and work tracking.
- Used Data Analysis Expressions DAX to create custom calculations in Power Pivot for Microsoft Excel workbooks and Analysis Services tabular model projects.
- Worked on Power BI reports using multiple types of visualizations including line charts, doughnut charts, tables, matrix, KPI, scatter plots, box plots, etc.
- Created Power BI reports more interact and activate by using storytelling features such as bookmarks, selection panels, drill through filters, etc.
- Created and administrated workspaces for each project on Power BI service and published the reports from Power BI Desktop to Power BI Services workspace.
- Utilized Power BI to create various analytical dashboards that helps business users to get quick insight of the data.
- Extract, Transform and Load data from Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, T-SQL, Spark SQL and U-SQL Azure Data Lake Analytics. Data Ingestion to one or more Azure Services - (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in Azure Databricks.
- Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform and load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse, write-back tool and backwards.
- Experience managing Azure Data Lakes (ADLS) and Data Lake Analytics and an understanding of how to integrate with other Azure Services.
- Migration of on premise data (Oracle/ Teradata) to Azure Data Lake Store(ADLS) using Azure DataFactory(ADF V1/V2).
- Created different types of triggers to automate the pipeline in ADF.
- Created, provisioned different Data-bricks clusters needed for batch and continuous streaming data-processing and installed the required libraries for the clusters.
Data Analyst
Capital One
02.2021 - 06.2023
- Automated infrastructure provisioning and maintenance using Terraform
- Reduced dev-hours for AWS resource maintenance including but not limited to provisioning and tagging
- Deployed microservices on AWS using ECS, EKS EC2, IAM, Lambda, S3, Route53, SNS, CloudWatch, RDS
- Validation & Review of Security Groups, IAM, S3, EC2 instances, and other AWS services to ensure secure implementation infrastructure
- Create and maintain CI/CD pipelines, integrating security, functional, and unit-testing for endpoint security tools, ensuring rapid and reliable deployment with minimal disruptions
- Utilized Cloud Watch to monitor resources such as EC2, CPU memory, Amazon RDS DB services, Dynamo DB tables
- Maintained and supported 70+ servers in multiple locations
- Designed a backup strategy that allowed the quick recovery of the servers and data they contained
- Worked on infrastructure with Docker containerization
- Collaborated with development support teams to set a continuous delivery environment with the use of Docker
- Written multiple Jenkins jobs to create and maintain AWS infrastructure across different AWS accounts
- Created scripts in Python for compliance validation and automated alerting and reporting of exceptions
- Log and event forwarding (System logs, CloudWatch, CloudTrail, AWS Config), and aggregation to Splunk Platform
- Automated the cloud deployment using AWS, AWS CLI, Python (pip, boto3), and AWS cloud formation templates
- Worked on Setup of JIRA Project, Issue type, Workflows, Custom fields, Field Configuration Schemes, Permission schemes, and Notification Schemes
- Automated deployments, scaling, and operations of application containers across clusters of hosts provided container-centric infrastructure by Kubernetes
- Upgraded major RDS Database Versions and performed Analysis of around 2500 tables using a parallel execution strategy involving AWS Lambda
- Worked with developing teams to support off-time hours to have successful deployments
- Supporting developing teams in major deployments and suggestions for resolutions of several incidents using emergency implementation plans
- Severed as the primary on-call and was responsible for all software, infrastructure, and firmware updates on the servers
- Create AWS Lambdas in Python to read the files in S3, import them to AWS DynamoDB
- Create and maintain network connectivity within AWS using AWS VPC (Virtual Private Cloud)
- Create different APIs to 'PUT,' 'POST,' and 'GET' data to and from the UI into the database and register them into the 'Exchange' service, which is a central registry of APIs for Capital One
- Create API gateway, AWS VPC, and AWS Route53 configurations for the web UI to interact with the AWS services
- Maintain the codebase in GIT and create GIT configuration for Continuous Integration (CI), kicking off the build processes, tests, and validations into different stages like test, pre-production, and production to run in Docker
- Create AWS Application Load Balancers (ALB) and manage the ALB Listeners and Target groups to validate the API requests automatically and scale AWS lambdas based on the API traffic
- Create AWS CloudFormation templates to provision all the infrastructure used in the application programmatically and store in GIT for versioning and collaboration
- Capture the API logs, traffic logs using Cloud Trail, VPC flow logs, and AWS resource logs and store them in AWS CloudWatch for alerting
- Export the logs to Splunk to aggregate the logs from AWS and internal Capital One applications and create dashboards in Splunk for a quick review of the application's health.
Data Engineer
DELL Technologies
04.2020 - 12.2020
- Involved in designing and deploying multi-tier applications using all AWS services like (EC2, Route53, S3, RDS, Dynamo DB, SNS, SQS, and IAM) focusing on high-availability, fault tolerance, and auto-scaling in AWS Cloud Formation
- Hands-on experience in resolving incident tickets related to Hadoop components like HBase, Yarn, Hive, and Kafka, and identifying root cause analysis
- Migrated an existing on-premises application to AWS
- Used AWS services like EC2 and S3 for small data sets processing and storage
- Experienced in maintaining a Hadoop cluster on AWS EMR
- Develop stored procedures/views in Snowflake and use them in Talend for loading Dimensions and Facts
- Worked in building ETL pipeline for data ingestion, data transformation, and data validation on cloud service AWS, working along with data steward under data compliance
- Worked on scheduling all jobs using Airflow scripts using Python and adding different tasks to DAG, and LAMBDA
- Used Pyspark for extracting, filtering, and transforming the data in data pipelines
- Skilled in monitoring servers using Nagios, Cloud Watch, and using ELK Stack Elasticsearch Kibana
- Used Data Build Tool for transformations in ETL processes, AWS Lambda, AWS SQS
- Worked on scheduling all jobs using Airflow scripts using Python
- Added different tasks to DAG and dependencies between the tasks
- Responsible for implementing monitoring solutions in Ansible, Terraform, Docker, and Jenkins
- Designed and implemented a test environment on AWS
- Responsible for Designing and configuring Network Subnets, Route Tables, Association of Network ACLs to Subnets, and Open VPN
- Responsible for Account Management, IAM Management, and Cost management
- Designed AWS Cloud Formation templates to create VPC, subnets, and NAT to ensure successful deployment of Web applications and database templates
- Creating S3 buckets also managing policies for S3 buckets and Utilizing S3 bucket and Glacier for storage and backup on AWS
- Experience to manage IAM users by creating new users, giving them limited access as per needs, assign roles and policies to specific users
- Refactored existing batch jobs and migrated existing legacy extracts from Informatica to Python based micro services and deployed in AWS with minimal downtime
- Designed and maintained databases using Python and developed Python based API (RESTful Web Service) using Flask
- Worked on a project driven on AWS Connect where we enabled communication between Lex, Lambda, and the customer.
Data Engineer
JP Morgan Chase
10.2017 - 01.2020
- Analyzed and gathered business requirements specifications by interacting with clients and understanding business requirement specification documents
- Involved in data analysis using Python and handling ad-hoc requests as per requirement
- Performed advanced procedures like text analytics and processing, using the in-memory computing capabilities of Spark using Python
- Worked on Migrating jobs from NiFi development to Pre-PROD and Production cluster
- Scheduled different Snowflake jobs using NiFi
- Used NiFi to ping Snowflake to keep Client Session alive
- Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs, and Scala, Python
- Develop a framework for converting existing PowerCenter mappings to PySpark (Python and Spark) Jobs
- Migrated applications to the PKS, GCP cloud
- Worked on google cloud platform (GCP) services like compute engine, cloud load balancing, cloud storage, cloud SQL, stack driver monitoring, and cloud deployment manager
- Create a Pyspark frame to bring data from DB2 to Amazon S3
- Translate business requirements into maintainable software components and understand the impact (Technical and Business)
- Provide guidance to the development team working on PySpark as an ETL platform
- Optimize the Pyspark jobs to run on Kubernetes Cluster for faster data processing
- Installed Oozie workflow engine to run multiple Hive and Pig jobs
- Written Hive Queries to fetch Data from HBase and transferred to HDFS through HIVE
- Developed Pig scripts in the areas where extensive coding needs to be reduced
- Worked with Spark Streaming to ingest data into the Spark engine
- Extensively used for all and bulk collect to fetch large volumes of data from a table
- Handled importing of data from various data sources using Sqoop, performed transformations using Hive, MapReduce, and loaded data into HDFS
- Created HBase tables to store various data formats of PII data coming from different portfolios
- Configured Sqoop and developed scripts to extract data from MySQL into HDFS
- Designed and configured Azure Virtual Networks (VNets), subnets, Azure network settings, DHCP address blocks, DNS settings, security policies, and routing
- Created a Docker file for each microservice and changed some of the Tomcat configuration files which are required to deploy Java-based applications to the Docker container
- Developed environments of different applications on AWS by provisioning EC2 instances using Docker, Bash, and Terraform
- Researched and implemented code coverage and unit test plugins with Maven/Jenkins
- Implemented Elastic Search on the Hive data warehouse platform
- Worked with ELASTIC MAPREDUCE and set up a Hadoop environment in AWS EC2 Instances
- Imported data from AWSS3 into Spark RDD, Performed transformations and actions on RDDs
- Good understanding of Cassandra architecture, replication strategy, gossip, snitch, etc
- Worked with the SCRUM team in delivering agreed user stories on time for every Sprint.
Python Developer
Kakatiya Solutions
07.2014 - 07.2017
- Assess the infrastructure needs for each application and deploy it on the Azure platform
- Developed tools using Python to automate some of the menial tasks
- Interfacing with supervisors, artists, systems administrators, and production to ensure production deadlines are met
- Used Python and Django to create graphics, XML processing, data exchange, and business logic implementation
- Utilized PyUnit, the Python unit test framework, for all Python applications
- Used Python-based GUI components for the front-end functionality such as selection criteria
- Designed and managed API system deployment using a fast HTTP server and Confidential AWS architecture
- Worked No Confidential Web Services (AWS) Cloud services such as EC2, EBS, S3, VPC, Cloud Watch, and Elastic Load Balancer
- Set up a database in AWS using RDS and configure backups for the S3 bucket
- Worked on Ad hoc queries, Indexing, Replication, Load balancing, and Aggregation in MongoDB
- Helped the big data analytics team with the implementation of Python scripts for Sqoop, Spark, and Hadoop batch Data Streaming
- Build and deployed the code artifacts into the respective environments in the Confidential Azure cloud
- Experience in process and transforming data by running USQL scripts on Azure
- Designed the user interface and client-side scripting using AngularJS framework, Bootstrap, and JavaScript
- Created User Interface Design using HTML5, CSS3, JavaScript, jQuery, JSON, REST and AngularJS, Bootstrap
- Developed GUI using JavaScript, HTML/HTML5, DOM, AJAX, CSS3, CQ5, and AngularJS in ongoing projects.
Education
Master of Science - Computer Technology
Eastern Illinois University
12.2018
Skills
- Cloud Platform: AWS, Azure, GCP
- Scripting/Programming: Shell scripting (ksh, Bash), Python, Java, SAS
- Version Control Tools: Git, GitHub, GitLab, Bit bucket
- CI Tools: Jenkins
- Databases: MS SQL Server 2008, MySQL, Redis, Cassandra, Oracle
- Container Tools: Docker, ECS
- Ticketing Tools: Jira, Bugzilla
- Build tools: Ant and Maven
Certification
AWS Certified Developer - Associate
Timeline
Data Engineer
AT&T
07.2023 - Current
Data Analyst
Capital One
02.2021 - 06.2023
Data Engineer
DELL Technologies
04.2020 - 12.2020
Data Engineer
JP Morgan Chase
10.2017 - 01.2020
Python Developer
Kakatiya Solutions
07.2014 - 07.2017
Master of Science - Computer Technology
Eastern Illinois University