Summary
Overview
Work History
Education
Skills
Certification
Timeline
Customer Champion Award
Generic

SAIKRISHNA BONALA

Scottsdale,AZ

Summary

Results-driven Data Engineer with over 13 years of experience in designing, building, and maintaining robust data pipelines and architectures on the Databricks platform. Expertise in constructing big data pipelines and ETL workflows using technologies such as Hadoop, Hive, HDFS, Spark, and cloud services including AWS S3, AWS EMR, Lambda, GLUE, and Redshift. Committed to leveraging advanced AWS capabilities and Databricks to process large-scale datasets while developing and deploying machine learning models that drive data-driven business growth. Eager to contribute to a dynamic team focused on innovative solutions and transformative data strategies.

Overview

13
13
years of professional experience
1
1
Certification

Work History

Sr. Data Engineer

Solugenix Corporation
India
11.2021 - Current
  • Architected an end-to-end ingestion pipeline for Cyber Self-Assessment data using Data bricks Lakehouse Platform (Unity Catalog, Data bricks SQL, Spark SQL, Python) to ingest raw data from mongoDB/PostgreSQL into a Medallion Architecture, utilizing Data bricks Autoloader for incremental file processing, Delta Live Tables (DLT) for automated transformation, and Data bricks Workflows with Git Repos for CI/CD orchestration, reducing ingestion latency by 60%.
  • Implemented Databricks AI/BI Genie on policy and claims datasets, empowering business stakeholders to generate instant insights via natural language, reducing ad-hoc SQL request volume by ~40%.
  • Curated a dedicated Genie Space for the underwriting team, enabling conversational analysis of risk profiles, premium variations, and policy retention rates.
  • Developed efficient Spark pipelines for cyber insurance data processing tasks and Implemented Delta Lake for data management and performance improvement.
  • Built and deployed machine learning models on Data bricks for insurance model, achieving significant improvements in risk prediction accuracy, streamlined policy pricing, and faster claims processing.
  • Automated data workflows using Data bricks jobs, leading to scalable data pipelines, and the ability to deliver timely insights for better decision-making.
  • Monitored and optimized Data bricks cluster performance, ensuring actively tracking the performance of Data bricks clusters to identify bottlenecks and inefficiencies and cost savings.
  • Designed and implemented serverless data pipelines using AWS Lambda, AWS Step Functions and AWS Glue to orchestrate complex workflows and process terabytes of data daily.
  • Developed a python script to transfer files from SFTP server to AWS S3 using the botocore and boto3 modules.
  • Developed a python script to check the status of the files at SFTP location like file availability, file name, header of the file, empty file, number of lines in the file at SFTP server. Script Automatically generates file status report and sends the file status email to source team if there are any anomalies in the file before ingesting data into AWS S3 location.
  • Worked in an Agile environment used Jira to track and manage user stories, epics, and tasks throughout the development lifecycle, and actively taken part in daily stand-ups, backlog grooming, and sprint planning.
  • Involved in creating hive tables, loading with AWS S3 data, and writing PySpark queries which will invoke and run spark jobs in the backend.
  • Developed a Data pipeline using Spark, Hive, Impala, and HBase to analyze customer behavioral data and financial histories in the Hadoop cluster.
  • Integrated NoSQL databases with big data technologies; Hadoop and Spark for seamless data processing and analysis.
  • Worked on NoSQL Databases MongoDB, and SPARK for real-time streaming of data into the cluster.
  • Expertise in processing JSON, Avro, Parquet, ORC, and CSV formats for efficient data ingestion, transformation, and storage.
  • Client: Marsh & McLennan companies, India
  • Environment: Hadoop, Hive, Spark SQL, HDFS, PySpark, AWS EMR, AWS S3, AWS Anthena, AWS Lambda, AWS Step and AWS GLUE, Amazon Redshift, Apache Flink, MongoDB, Databricks, Informatica PowerCenter, Informatica BDM, RDBMS, MySQL, Sqoop, Spark-SQL, Oracle and UNIX.

Big Data Engineer

TATA Consultancy Services, India
India
10.2015 - 11.2021
  • Designed and implemented serverless data pipelines using AWS Lambda, AWS Step Functions and AWS Glue to orchestrate complex workflows and process terabytes of data daily.
  • Designed and created Hive external tables using shared meta-store with static partitioning, dynamic partitioning, and bucketing.
  • Responsible for managing data coming from different sources and involved in HDFS maintenance and loading of structured and semi-structured data.
  • Involved in creating hive tables, loading with data, and writing hive queries which will invoke and run map reduce jobs in the backend.
  • Written Spark applications using Python on Azure Databricks to interact with the MySQL database using Spark SQL context and accessed Hive tables using Hive context.
  • Developed PySpark scripts to perform complex transformations, ensuring the seamless integration of diverse data sources.
  • Complete understanding of customer spending patterns, customers segments based on their profiles, customer preferences, AQB, AQB, (Average Quarterly Balance), MAB (Monthly Average Balance) etc.
  • Understanding the customer DNA mapping document provided by Business Analyst and logic clarifications from DM team (Client) to understand the dataflow.
  • Developed and deployed Apache Spark applications on Databricks for large-scale data processing and analytics.
  • Worked on designing, implementing, and maintaining MySQL, Oracle, PostgreSQL, and MongoDB databases.
  • Create Azure Blob Storage for Import/Export data to/from .CSV File.
  • Developed spark scripts by using python shell commands as per the requirement.
  • Used Avro, Parquet data formats to store into HDFS.
  • Extracted terabyte-scale IoT and transactional data from on-prem sources, establishing incremental replication into Azure Data Factory.
  • Established a foundation for real-time analytics with Event Hubs to process millions of events per second into a data platform with near-zero latency or data loss.
  • Loaded batch data into Delta Lake, enabling reliability, performance, and interoperability across the Azure analytics engines.
  • Implemented Agile methodology to streamline project delivery, fostering collaboration between cross-functional teams and ensuring efficient development cycles.
  • Client: CITI Bank, USA
  • Environment: Hadoop, Hive, Spark SQL, HDFS, PySpark, Azure Data Factory, Azure Data Lake Storage, Azure Synapse Analytics, Azure Databricks, Delta Lake, Azure SQL Database, GitHub, Azure DevOps, AWS EMR, AWS S3, AWS Anthena, AWS Lambda, AWS Step and AWS GLUE, Amazon Redshift, Apache Flink, MongoDB.

Data Analyst

Equifax, India
India
04.2013 - 10.2015
  • Designed and created Hive external tables using shared meta-store with static partitioning, dynamic partitioning, and bucketing.
  • Responsible for managing data coming from different sources and involved in HDFS maintenance and loading of structured and semi-structured data.
  • Designed and developed applications on the data lake to transform the data according to business users to perform analytics.
  • Developed modules to see the trend of credit cards on various key factors such as Spends, Balance, Acquisition, Validation checks, Execution.
  • Analyze the trend of credit card customers on various aspects like CIF (Card in force), New cards opened, Spends, ATS, Outstanding.
  • Analyze the trend of credit card customers by their Product (Domestic, Salary, NRI), Segment (Privee, Wealth, Priority, Prime, Prime Plus).
  • Happened to work on different files like CSV, txt, and fixed width to load data from various sources to raw tables.
  • Conducted data model reviews with team members and captured technical metadata through modeling tools.
  • Implemented ETL process wrote and optimized SQL queries to perform data extraction and merging from SQL server database.
  • Extracted data from RDBMS using Sqoop and stored it in HDFS for further processing.
  • Applied data visualization techniques and designed interactive dashboards using SAS Visual Analytics to present complex reports, charts, summaries, and graphs to team members and stakeholders.
  • Administered and maintained HDFS using the Hadoop-Java API.
  • Analyzed data using Spark and Hive, generating insightful summary results for downstream systems.
  • Involved in designing and developing tables in HBase and storing aggregated data from Hive tables.
  • Developed complex MapReduce jobs for performing efficient data transformations.
  • Strong Experience in writing SQL queries.
  • Responsible for triggering the jobs using AutoSys.
  • Client: AXIS Bank
  • Environment: Hadoop, Hadoop Distributed File System, MapReduce, Hive, Spark, Sqoop, Power BI, Shell Script, SQL, AutoSys, BASE SAS, SAS MACROS, SAS Enterprise guide.

Education

Bachelor 's - Electronics and Communications

JNTUH India
India
05-2012

Skills

  • Databricks Stack: Delta Lake, Unity Catalog, MLflow, Databricks SQL, Medallion Architecture (Bronze/Silver/Gold)
  • AWS cloud services expertise
  • Generative AI & Productivity: Microsoft Copilot, ChatGPT
  • Hadoop Distribution: Cloudera, Horton Works
  • Big Data Technologies: HDFS, MapReduce, Hive, Spark streaming, Spark, Sqoop
  • Languages: SQL, PL/SQL, Python, HiveQL, PySpark, SAS
  • ETL/ Datawarehouse Tools: Informatica PowerCenter, Informatica BDM
  • Data Analytics Tools: BASE SAS and SAS MACROS
  • Visualization Tools: Tableau, SAS Visual analytics
  • Operating Systems: Windows (XP/7/8/10), UNIX, LINUX
  • Version Control Tools & CI/CD: GIT, GitHub, Jenkins
  • Databases: MS SQL Server, MS Excel, MS Access, Oracle 11g/12c
  • Experienced with NoSQL database management
  • File Formats: JSON, CSV, ORC, Parquet, Avro
  • Data processing pipeline development
  • Data modeling

Certification

1.AWS Certified Data Engineer - Associate

2.SAS Certified Professional: Advanced Programming Using SAS 9.4

Timeline

Sr. Data Engineer

Solugenix Corporation
11.2021 - Current

Big Data Engineer

TATA Consultancy Services, India
10.2015 - 11.2021

Data Analyst

Equifax, India
04.2013 - 10.2015

Bachelor 's - Electronics and Communications

JNTUH India

Customer Champion Award

1. Customer champion Award

2. Empathy in Action Awards

3. PAT on the Back Awards

4. Star Performer Award

SAIKRISHNA BONALA