Project
Advanced Distributed Data Systems Lab
A reproducible engineering portfolio spanning Hadoop, HBase, Spark, streaming, and classification.
Hadoop · Spark · HBase · Spark Streaming · Python · Java
Product Snapshot
- Role
- Independent coursework engineering
- Stage
- Code and reports completed; metrics apply only to the documented course environments.
- Focus
- Distributed computing, inverted indexes, stream processing, HBase versions, performance tuning, and behavior prediction.
- Validation
- Completed a three-node Hadoop setup, Spark search and aggregation, HBase APIs, social-stream processing, and classification tuning.
- Public Proof
- Signed lab reports, code, run captures, and metric records; the prediction lab explicitly used schema-compatible synthetic data.
Next Step
Why this is one project
These labs form a path from offline batch processing to streams, storage APIs, and model tuning. I grouped them as one traceable engineering portfolio rather than presenting disconnected assignments.
Batch processing and retrieval
On a three-node Huawei Cloud Hadoop environment, I processed roughly 629 million bytes of text: 5,022,247 lines, 80,358,163 tokens, and 75,335,916 transitions. A separate MapReduce request analysis handled 37,449,160 records.
The Spark work includes aggregation over 12.6 million orders and a simplified THUCNews search engine using TF-IDF and an inverted index.
Storage and streaming
HBase labs covered shell operations, Java APIs, historical versions, and version-value calculations. A Spark Streaming exercise processed 120,000 of 422,314 source social posts at 10,000 records per second, using a 30-second window and 10-second slide across 1,752 tag types.
Model and performance tuning
A behavior-prediction lab used 120,000 training and 100,000 stream records and recorded AUC 0.7156. Tuning reduced driver processing time from 431 ms to 202 ms with unchanged accuracy.
Because the original course dataset was unavailable, this experiment used schema-compatible synthetic data. That limitation is part of the result, not a footnote.
Takeaway
The work taught me to treat data cutoff, execution environment, metric definition, and source provenance as part of an engineering result.
Configure the public Giscus environment variables to open discussion on the public site.