Danpei Jiang
Troy,MI
Overview
9
9
years of professional experience
Work History
Big Data Engineer
General Motors Co
Warren, MI
05.2016 - Current
- Implemented real-time streaming applications using Kafka Streams or Spark Streaming.
- Used statistical software to analyze and process large data sets.
- Analyzed large datasets to identify patterns and trends in data.
- Created graphs and charts detailing data analysis results.
- Built data pipelines using Apache Spark to process large amounts of structured and unstructured data.
- Tested, validated and reformulated models to foster accurate prediction of outcomes.
- Developed automated processes for collecting, organizing, and analyzing big data.
- Implemented best practices around designing fault tolerant architectures in a distributed computing environment.
- Developed ETL jobs to extract, transform, and load data from various sources into the target system.
- Optimized queries on distributed systems like Hive and Impala or Presto to improve performance of analytics tasks.
- Performed end-to-end testing of all components in the Big Data architecture.
- Identified areas where automation can be used to streamline development workflows.
- Automated deployment processes for deploying applications onto different cloud environments.
- Used scripting languages such as Python or Bash to automate routine tasks associated with Big Data engineering.
- Provided technical support to business users on using Big Data tools for their analytical needs.
- Cleaned and manipulated raw data.
- Developed new functions and applications to conduct analyses.
- Collaborated with other members of the team to design efficient solutions for managing Big Data workloads.
- Collaborated closely with team members to achieve project objectives and meet deadlines.
Education
Master of Arts - Computer Science
University of South Dakota
Vermillion, SD05-2018
Master of Arts - Mathematics
University of South Dakota
Vermillion, SD05-1996
Skills
- Scala programming
- Real-time processing
- Agile methodology
- Apache flink
- Microsoft Azure
- Data modeling
- Data migration
- Java programming
- Continuous deployment
- Data pipeline design
Timeline
Big Data Engineer
General Motors Co
05.2016 - Current
Master of Arts - Computer Science
University of South Dakota
Master of Arts - Mathematics
University of South Dakota
Similar Profiles
Michael Van HornMichael Van Horn
Senior Finance Lead - Accessory Pricing at General Motors CoSenior Finance Lead - Accessory Pricing at General Motors Co
Taylor ClaytonTaylor Clayton
Machine Learning Data Scientist at General Motors CoMachine Learning Data Scientist at General Motors Co
Hadi H DaanaHadi H Daana
Software Developer at General Motors CoSoftware Developer at General Motors Co
Jerry JordanJerry Jordan
Project Manager at General Motors CoProject Manager at General Motors Co
JESSICA LEWISJESSICA LEWIS
Test Development Specialist at Independent ContractorTest Development Specialist at Independent Contractor