Danpei Jiang
Troy,MI
Overview
9
9
years of professional experience
Work History
Big Data Engineer
General Motors Co
Warren, MI
05.2016 - Current
- Implemented real-time streaming applications using Kafka Streams or Spark Streaming.
- Used statistical software to analyze and process large data sets.
- Analyzed large datasets to identify patterns and trends in data.
- Created graphs and charts detailing data analysis results.
- Built data pipelines using Apache Spark to process large amounts of structured and unstructured data.
- Tested, validated and reformulated models to foster accurate prediction of outcomes.
- Developed automated processes for collecting, organizing, and analyzing big data.
- Implemented best practices around designing fault tolerant architectures in a distributed computing environment.
- Developed ETL jobs to extract, transform, and load data from various sources into the target system.
- Optimized queries on distributed systems like Hive and Impala or Presto to improve performance of analytics tasks.
- Performed end-to-end testing of all components in the Big Data architecture.
- Identified areas where automation can be used to streamline development workflows.
- Automated deployment processes for deploying applications onto different cloud environments.
- Used scripting languages such as Python or Bash to automate routine tasks associated with Big Data engineering.
- Provided technical support to business users on using Big Data tools for their analytical needs.
- Cleaned and manipulated raw data.
- Developed new functions and applications to conduct analyses.
- Collaborated with other members of the team to design efficient solutions for managing Big Data workloads.
- Collaborated closely with team members to achieve project objectives and meet deadlines.
Education
Master of Arts - Computer Science
University of South Dakota
Vermillion, SD05-2018
Master of Arts - Mathematics
University of South Dakota
Vermillion, SD05-1996
Skills
- Scala programming
- Real-time processing
- Agile methodology
- Apache flink
- Microsoft Azure
- Data modeling
- Data migration
- Java programming
- Continuous deployment
- Data pipeline design
Timeline
Big Data Engineer
General Motors Co
05.2016 - Current
Master of Arts - Computer Science
University of South Dakota
Master of Arts - Mathematics
University of South Dakota
Similar Profiles
Okeshia JusticeOkeshia Justice
Team Leader at General Motors CoTeam Leader at General Motors Co
Bruce RyanBruce Ryan
Material Operator at General Motors CoMaterial Operator at General Motors Co
James ButterworthJames Butterworth
Engineering Group Manager at General Motors CoEngineering Group Manager at General Motors Co
Steven DyeSteven Dye
Team Leader at General Motors CoTeam Leader at General Motors Co
Shelley MartinShelley Martin
Quality Control Inspector at Stellantis - MoparQuality Control Inspector at Stellantis - Mopar