Vijay Kumar Tulluri

• Engineered and executed stream processing jobs using Spark Streaming in Scala, enhancing real-time data analysis capabilities.
• Developed real-time data processing applications in Scala and Java, integrating Apache Spark Streaming with Kafka and JMS for efficient data ingestion.
• Utilized PySpark and Hive for sensor data analysis, clustering users based on behavioral patterns during events, showcasing deep analytical skills.
• Implemented high-performance, real-time processing jobs using Spark Streaming, coupled with Kafka, to establish a robust data pipeline system.
• Leveraged AWS services like EMR and EC2 for scalable and efficient Big Data processing, demonstrating cloud computing expertise.
• Crafted and optimized Spark programs, creating data frames and performing transformations to facilitate complex data analysis tasks.
• Designed and managed data ingestion from diverse sources using Kafka producers and partitions, applying custom encoders for efficient data loading.
• Conducted proof-of-concept (POC) with Hadoop, extracting data into HDFS using Spark, showcasing innovation and technical prowess.
• Optimized data processing and analytics using Spark SQL and Scala on Cloudera Hadoop YARN, enhancing data insights extracted from Hive.

Developed and automated Databricks notebooks using SQL and Python, configuring Azure Databricks clusters for high concurrency, ensuring rapid preparation of high-quality data.

Environment: Hadoop, Spark, Spark Streaming, MapReduce, Hive, Pig, Oozie, Kafka, Storm, Scala, Java, Python, Sqoop, Talend, AWS (EMR, S3, CloudWatch), MongoDB, Solr, Hadoop Cluster, Azure Databricks, Linux.

• Analyzed and optimized large datasets, leveraging Azure Cloud services and Spark for efficient data aggregation and reporting.
• Designed and implemented a scalable Big Data Analytics architecture, integrating technologies like ADLS Gen-2, Delta Lakes, and Hive.
• Developed robust Spark applications in Scala and Python for batch data extraction, transforming source data into parquet format for storage in Azure Data Lake Services (ADLS Gen 2).
• Constructed ETL pipelines using Databricks notebooks, integrating wear algorithm predictions with outputs to Azure CosmosDB for real-time analytics.
• Engineered Kafka consumers in Scala to ingest data from Kafka topics, enhancing data flow and accessibility within the pipeline.
• Migrated legacy HQL computational code to PySpark, streamlining data processing workflows and improving system performance.
• Facilitated data ingestion from various sources into Azure Data Storage services, employing Azure Data Factory, T-SQL, and Spark SQL for comprehensive data management.
• Deployed Spark jobs on Amazon EMR, leveraging AWS cluster computing for scalable data processing and analysis.
• Integrated data engineering practices with the Tire Wear model, collaborating with the data science team to refine predictive outputs and resolve technical issues.
• Implemented data warehouse solutions on AWS Redshift, leading projects to migrate on-premises databases to cloud-based storage and computing environments.

Environment: Hadoop, Hive, Spark, MapReduce, HBase, Kafka, Flume, Azure Databricks, AWS, Azure Data Lake, Azure SQL, Azure DW, Scala, Python, Java, Shell Scripting

HDFS, ADLS Gen-2, Cassandra, Teradata, NoSQL, Sqoop, Ambari, Azure Data Factory, Tableau, Ubuntu, Oracle 10g/11g/12C

• Designed and implemented complex T-SQL queries, stored procedures, and tables for data extraction and manipulation to meet business requirements.
• Developed and executed shell scripts for automating DataStage jobs, enhancing efficiency and reliability in data processing workflows.
• Enhanced data accessibility for service developers by identifying and cataloging essential content within existing reference models.
• Leveraged Spark SQL API within PySpark for efficient data extraction, loading, and execution of SQL queries for analytical processing.
• Crafted PySpark scripts incorporating hashing algorithms to encrypt sensitive data, ensuring data privacy and compliance with client specifications.
• Engineered Python-based APIs (RESTful Web Services) for revenue tracking and analysis, providing critical insights into financial performance.
• Optimized SQL queries through meticulous analysis of indexes and execution plans, significantly reducing runtime and enhancing database performance.
• Conducted comprehensive ETL testing, overseeing job execution, data extraction, transformation, and loading into Data Warehouse servers, ensuring data integrity and accuracy.
• Spearheaded data migration projects utilizing a blend of SQL, Azure Data Factory, SSIS, and PowerShell, achieving seamless transition to Azure Data Storage services.
• Created dynamic dashboards and ad-hoc reports using Tableau and Power BI, facilitating data-driven decision-making by visualizing POS and revenue data.

Environment: Spark, PySpark, AWS, S3, Glue, Redshift, DynamoDB, Hive, Spark SQL, Docker, Kubernetes, Airflow, GCP, ETL workflows.