Nithya Santhoshini Chanda

Responsibilities:

• Designed and implemented Enterprise Data Lake supporting various use cases, including storage, processing, analytics, and reporting of large, rapidly changing data, utilizing avariety of AWS services.
• Established Kinesis Data streams, Kinesis Data Firehose, and Kinesis Data Analytics to capture and process streaming data, subsequently outputting it into S3, Dynamo DB, and Redshift for storage and analysis.
• Developed ETL integration patterns using Python on Spark for efficient data processing.
• Created aframework for converting existing PowerCenter mappings to PySpark (Python and Spark) Jobs.
• Optimized volumes and EC2 instances, creating multiple Virtual Private (VP) instances, and managed IAM to establish new accounts, roles, and groups.
• Implemented SparkRDD transformations to map business analysis and apply actions on top of transformations.
• Built S3 buckets, managed policies for S3 buckets, and utilized S3 bucket and Glacier for storage and backup on AWS.
• Enhanced and tuned Redshift environment, achieving query performance improvements of up to 100x for Tableau and SAS Visual Analytics.
• Integrated services such as GitHub, AWS Code Pipeline, Jenkins, and AWS Elastic Beanstalk to create a streamlined deployment pipeline.
• Established monitors, alarms, and notifications for EC2 hosts using CloudWatch and implemented new project builds framework using Jenkins as abuild tool.
• Designed and implemented ETL jobs for data processing using various AWS services, including Glue Crawlers for automatic schema discovery, S3 buckets for storage, and Athena for data transformation, resulting in a 90% reduction in processing time.
• Extracted data from multiple source systems (S3, Redshift, RDS) and created tables/databases in Glue Catalog by utilizing Glue Crawlers.
• Wrote code to optimize the performance of AWS services used by application teams and provided code-level application security for clients (IAM roles, credentials, encryption, etc.).
• Designed and created a serverless AWS Glue ETL pipeline using Scala to transform and load data from S3 buckets into Amazon Redshift, utilizing CloudWatch events to trigger scheduled Lambda functions for automated ETL processes.
• Utilized Athena for data analysis and QuickSight to generate business intelligence reports from processed data obtained through Glue ETL Jobs.
• Converted and delivered 300M+ CSV records into 3.5M Parquet files using AWS Glue Jobs and Python scripts.
• Redesigned architecture and changed the triggering of long-running ETL jobs from Lambda functions to Workflows.
• Designed dashboards for the Code Quality team to gain insights into developer productivity using Metabase and Python.
• Developed a generic framework using Scala and bash scripts for data ingestion, processing different data formats (CSV, JSON, PARQUET), and feeding the data into Hive tables.
• Implemented Spark and Python scripts to fetch build data and test results from Jenkins, pushing them into AWS S3 and Redshift for faster reporting.
• Developed and maintained Directed Acyclic Graphs (DAGs) in Apache Airflow, defining the sequence and dependencies of tasks for efficient data pipeline execution.
• Created custom operators in Python for specialized tasks, extending the functionality of Apache Airflow to seamlessly integrate with unique data processing requirements.· Designed and implemented an Enterprise Data Lake supporting various use cases, including the storage, processing, analytics, and reporting of large, rapidly changing data, utilizing a variety of AWS services.
• Established Kinesis Data streams, Kinesis Data Firehose, and Kinesis Data Analytics to capture and process streaming data, subsequently outputting it into S3, Dynamo DB, and Redshift for storage and analysis.
• Developed ETL integration patterns using Python on Spark for efficient data processing.
• Created a framework for converting existing PowerCenter mappings to PySpark (Python and Spark) Jobs.
• Optimized volumes and EC2 instances, creating multiple Virtual Private (VP) instances, and managed IAM to establish new accounts, roles, and groups.
• Implemented SparkRDD transformations to map business analysis and apply actions on top of transformations.
• Built S3 buckets, managed policies for S3 buckets, and utilized S3 bucket and Glacier for storage and backup on AWS.
• Enhanced and tuned the Redshift environment, achieving query performance improvements of up to 100x for Tableau and SAS Visual Analytics.
• Integrated services such as GitHub, AWS Code Pipeline, Jenkins, and AWS Elastic Beanstalk to create a streamlined deployment pipeline.
• Established monitors, alarms, and notifications for EC2 hosts using CloudWatch and implemented a new project builds framework using Jenkins as a build tool.
• Designed and implemented ETL jobs for data processing using various AWS services, including Glue Crawlers for automatic schema discovery, S3 buckets for storage, and Athena for data transformation, resulting in a 90% reduction in processing time.
• Extracted data from multiple source systems (S3, Redshift, RDS) and created tables/databases in Glue Catalog by utilizing Glue Crawlers.
• Wrote code to optimize the performance of AWS services used by application teams and provided code-level application security for clients (IAM roles, credentials, encryption, etc.).
• Designed and created a serverless AWS Glue ETL pipeline using Scala to transform and load data from S3 buckets into Amazon Redshift, utilizing CloudWatch events to trigger scheduled Lambda functions for automated ETL processes.
• Utilized Athena for data analysis and QuickSight to generate business intelligence reports from processed data obtained through Glue ETL Jobs.
• Converted and delivered 300M+ CSV records into 3.5M Parquet files using AWS Glue Jobs and Python scripts.
• Redesigned architecture and changed the triggering of long-running ETL jobs from Lambda functions to Workflows.
• Designed dashboards for the Code Quality team to gain insights into developer productivity using Metabase and Python.
• Developed a generic framework using Scala and bash scripts for data ingestion, processing different data formats (CSV, JSON, PARQUET), and feeding the data into Hive tables.
• Implemented Spark and Python scripts to fetch build data and test results from Jenkins, pushing them into AWS S3 and Redshift for faster reporting.
• Developed and maintained Directed Acyclic Graphs (DAGs) in Apache Airflow, defining the sequence and dependencies of tasks for efficient data pipeline execution.
• Created custom operators in Python for specialized tasks, extending the functionality of Apache Airflow to seamlessly integrate with unique data processing requirements.

Environment:

Python, PySpark, Spark, Scala, AWS Cloud Data, Amazon EMR, Amazon S3 ETL(Informatica), Lambda, Glue Crawlers, CloudWatch, Workflows, Kafka, Hadoop (HDP 3.0), Tableau, Cloudera, Postgres, Teradata, Terraform, Splunk, AWS Redshift, Airflow

Responsibilities:

• Analyzed results using Spark SQL queries in conjunction with Hive queries to derive meaningful insights from large datasets.
• Implemented Spark with Scala and Spark SQL to optimize data processing performance, enabling faster testing and processing of data.
• Utilized Spark with Scala, leveraging DataFrames and Spark SQL API to expedite the processing of data.
• Designed and created HBase tables, employing column families to efficiently store and manage project-related data.
• Employed Maven for deployments and managed structured, semi-structured (e.g., XML), and unstructured data in the data lake with a streamlined data flow.
• Established connectors to facilitate data loading to and from Snowflake, utilizing analysis to discover and align with business goals.
• Diagnosed issues and provided business intelligence for Hadoop through the effective utilization of Splunk.
• Visualized data collected from diverse sources using Splunk, enhancing the understanding of data patterns and trends.
• Demonstrated experience in developing/consuming Web Services (REST, SOAP, JSON) and APIs within Service-oriented architectures.
• Constructed Chef-based CI/CD solutions, improving developer productivity and enabling rapid deployments.
• Troubleshooted Linux network and security-related issues, utilizing tools like IP tables and firewall, and captured packets for analysis.
• Extensively worked on importing metadata into Hive, migrating existing tables and applications onto Hive for enhanced data management.
• Leveraged Tableau to create compelling stories and interactive dashboards, offering detailed insights from the analyzed data.
• Wrote various Spark transformations in Scala to cleanse, validate, and summarize user behavioral data.
• Developed a production planning dashboard in Spotfire by creating information links to seamlessly integrate data sources.
• Identified anomalies and new data sources for analysis, enhancing data collection and analysis flow into the data lake.
• Conducted data wrangling using the Pandas library for cleaning, transforming, and reshaping data.
• Utilized R and Python for Exploratory Data Analysis, comparing and assessing the effectiveness of data in the data lake.
• Generated statistics from analyzed data, producing insightful reports for informed decision-making.
• Parsed unstructured data into a semi-structured format by implementing complex algorithms in Spark.
• Loaded transformed data into Hive tables, enabling analytics and insights using Hive queries.
• Implemented partitioning on Hive data to enhance performance in data processing and flow.
• Analyzed data by executing Hive queries (Hive QL) to gain insights into customer behavior.
• Worked on various performance optimizations in Hive, including the use of distributed cache for small datasets and partitioning and bucketing strategies.
  

Environment:

AWS (IAM, EC2, S3, EBS, Glacier, ELB, Cfn, CloudWatch, Cloud Trail, Hadoop (HDP), Informatica, HDFS, Spark SQL, Git, Kafka, Hive, Java, Scala, HBase, Maven, UNIX Shell Scripting, AWS Cloud Data, Terraform, Python, SQL, Tableau, Pig, Teradata, Splunk

Responsibilities:

• Developed and maintained SQL scripts for data extraction, transformation, and loading (ETL) processes.
• Automated repetitive tasks using Python scripts, improving overall efficiency in daily workflows.
• Established and documented coding standards to ensure consistency and code quality within the development team.
• Conducted code reviews and provided constructive feedback to enhance the overall codebase.
• Implemented version control using Git, ensuring collaborative and streamlined development processes.
• Integrated unit testing into the development workflow, enhancing code reliability and reducing bugs.
• Collaborated with cross-functional teams to gather and analyze requirements, ensuring alignment with project goals.
• Contributed to the creation of RESTful APIs, facilitating seamless communication between different components of the application.
• Utilized Docker for containerization, enabling consistent deployment across various environments.
• Implemented monitoring and logging solutions to track system performance and troubleshoot issues effectively.
• Collaborated with DevOps teams to optimize continuous integration and continuous deployment (CI/CD) pipelines.
• Actively participated in Agile development methodologies, including Scrum meetings and sprint planning.
• Developed and maintained technical documentation, ensuring comprehensive knowledge transfer within the team.
• Engaged in ongoing professional development, staying updated on industry best practices and emerging technologies.
• Conducted training sessions for team members on best coding practices and the use of new tools and technologies.
• Established error-handling mechanisms in scripts and applications, enhancing data processing reliability.
• Assisted in the design and implementation of a robust security framework for web applications.
• Implemented caching strategies to optimize data retrieval and processing times.
• Collaborated with UX/UI designers to implement visually appealing and user-friendly interfaces.
• Contributed to the creation of data models, ensuring database structures aligned with application requirements.
• Conducted performance testing to identify and address bottlenecks in applications.
• Collaborated with the Quality Assurance (QA) team to address and resolve reported issues promptly.
• Collaborated with the user support team to address and resolve customer-reported issues.
• Implemented A/B testing strategies to optimize user experience and feature effectiveness.
• Actively participated in technology meetups and conferences to stay updated on industry trends.
• Engaged in cross-functional retrospectives to identify areas for process improvement.
• Implemented data anonymization techniques for compliance with privacy regulations.

Environment:

Python, Django, Mongo DB, Selenium, Pandas, Java, J Query, Zookeeper, bootstrap, My SQL, Linux, Ajax, JavaScript, Apache, JIRA, Cassandra, HTML5 and CSS, Angular JS, Backbone JS