Danpei Jiang
Troy,MI
Overview
9
9
years of professional experience
Work History
Big Data Engineer
General Motors Co
Warren, MI
05.2016 - Current
- Implemented real-time streaming applications using Kafka Streams or Spark Streaming.
- Used statistical software to analyze and process large data sets.
- Analyzed large datasets to identify patterns and trends in data.
- Created graphs and charts detailing data analysis results.
- Built data pipelines using Apache Spark to process large amounts of structured and unstructured data.
- Tested, validated and reformulated models to foster accurate prediction of outcomes.
- Developed automated processes for collecting, organizing, and analyzing big data.
- Implemented best practices around designing fault tolerant architectures in a distributed computing environment.
- Developed ETL jobs to extract, transform, and load data from various sources into the target system.
- Optimized queries on distributed systems like Hive and Impala or Presto to improve performance of analytics tasks.
- Performed end-to-end testing of all components in the Big Data architecture.
- Identified areas where automation can be used to streamline development workflows.
- Automated deployment processes for deploying applications onto different cloud environments.
- Used scripting languages such as Python or Bash to automate routine tasks associated with Big Data engineering.
- Provided technical support to business users on using Big Data tools for their analytical needs.
- Cleaned and manipulated raw data.
- Developed new functions and applications to conduct analyses.
- Collaborated with other members of the team to design efficient solutions for managing Big Data workloads.
- Collaborated closely with team members to achieve project objectives and meet deadlines.
Education
Master of Arts - Computer Science
University of South Dakota
Vermillion, SD05-2018
Master of Arts - Mathematics
University of South Dakota
Vermillion, SD05-1996
Skills
- Scala programming
- Real-time processing
- Agile methodology
- Apache flink
- Microsoft Azure
- Data modeling
- Data migration
- Java programming
- Continuous deployment
- Data pipeline design
Timeline
Big Data Engineer
General Motors Co
05.2016 - Current
Master of Arts - Computer Science
University of South Dakota
Master of Arts - Mathematics
University of South Dakota