Jiani Tian

Sunnyvale,CA

Overview

years of professional experience

Work History

Senior Software Engineer

Intuit

Mountain View, California

01.2022 - Current

Designed, launched, and operated a production AI oncall assistant that has been running for 2+ years, autonomously triaging 100+ Kubernetes-hosted data pipeline failures daily across 35 data engineering teams and 7 Slack channels — built on a real-time RAG pipeline chaining BPP metadata, Splunk log discovery, and S3 EMR log analysis, deployed on IKS (Intuit kubernetes) with reliability hardening including deadlock prevention, bounded parsing guards, and token auto-refresh across AWS, Splunk, and Intuit's LLM serving infrastructure.
Built an internal developer platform that automatically generates data pipeline documentation from QuickETL (Intuit In-house ETL framework) configs — parsing SQL for table-level lineage, constructing data flow DAGs, and enriching with metadata from AWS, Tidal, Jira, and PagerDuty via a plugin architecture deployed across EMR clusters.
Designed for extensibility (MkDocs plugin lifecycle, Jinja2 templating, cross-account AWS integration) and runtime resilience (version-adaptive SQL parsing, caching, defensive error handling across heterogeneous compute environments).
Built ADAPT (Automated Data Pipeline Testing), a distributed pipeline testing platform (Scala/Spark + Java/Spring Boot on EKS) that automates dependency graph resolution, UAT environment generation, and end-to-end validation for 100+ ETL pipelines running on EMR Serverless, S3, and Hive turning multi-day manual regression cycles into single-trigger automated runs.
Led architecture spanning Spark job orchestration, DynamoDB state management, BPP scheduler integration, and config-driven table lineage extraction across multiple AWS accounts.

Senior Data Engineer

Healthcrowd, LLC

San Mateo, CA

08.2019 - 01.2022

Built and maintained the internal ETL project with Python to automate and optimize the data processing; Design the metrics for monitoring the periodic data analysis workflow for all the clients
Lead the internal ETL project using Spark and EC2 for data pulling and reporting, improve the efficiency by 50%
Implemented, launched, tested and maintained various campaigns (over 10 Health Plan clients and outreached over millions of customers) business logic in PHP with Symfony framework
Applied statistical methods including hypothesis testing on campaign metrics in Python to analyze campaign performance, designed and developed two web based apps using Python(Django) running on Linux system to guide clients with insights and recommendations to meet their ROI objects
Collaborated with teams including Software Development, Customer Success Management and Sales to convert the customers’ requirements into product specifications

Data and Policy Analyst II, Statistical Programmer

Acumen, LLC

Burlingame, CA

05.2017 - 08.2019

Performed advanced predictive models in R, such as cox and logistic regression; Random Forest, XGboost and Lasso for variable selection
Forecast the future trend of the drugs using BP neural network and gray model on Matlab; tune parameters by cross validation
Conducted hypothesis testing and gave summary statistics to FDA weekly; Visualized data in R and Python and implemented real time charts with HTML5 and CSS
Queried, cleaned and analyzed billion-lined claim and payment database by SQL, R and SAS
Programmed in SAS and R to build company internal code library

Education

Master of Arts - Statistics

Columbia University

New York, NY

12.2016

Bachelor of Science - Applied Mathematics

Tianjin University

Tianjin, TJ

07.2014

Skills

Programming Languages: Python Scala Java Bash AWK R SAS SQL Matlab
Big Data Tools: Spark AWS services (EMR, EC2, S3, DynamoDB, Redshift, CloudFormation)

Web development: Twig HTML CSS JavaScript, databases (MySQL, PostgreSQL, MongoDB)

Websites

Timeline