Why revisit an old project?
My original solution treated the Kaggle Disaster Tweets task as a conventional text-classification problem. I normalized the text, extracted TF-IDF features, and compared logistic regression with random forests. It was useful work at the time, but most of the difficult decisions were delegated to established libraries.
The revisit had a different objective: build a rigorous pipeline, verify the mechanics at every stage, and use the same dataset as a measurement of how my engineering instincts had changed.
Phase 1: a baseline I could explain completely
Before reaching for a pre-trained model, I built the data path and classifier with standard PyTorch utilities. The vocabulary explicitly reserved at index 0 and at index 1. Token embeddings were mean-pooled from (B, T, C) to (B, C), then passed to a linear classification head.
I checked the model before trusting the training loop. A randomly initialized binary classifier should begin near -ln(0.5) ≈ 0.693; this model began at 0.6734. I then forced it to overfit one batch. It reached 100% accuracy and a loss of 0.0103, confirming that the forward pass, gradients, optimizer, and labels were aligned.
The most valuable improvement was not a larger model. It was learning to prove that a small model and its training loop were correct before adding complexity.
Phase 2: restoring sequence information
Mean pooling is intentionally simple, but it discards word order. I replaced the baseline with a single Transformer encoder block and learned positional embeddings. The positional tensor was broadcast across the batch and added to the token representation before attention.
That sequence-aware model reached 78.07% validation accuracy. More importantly, it made the improvement attributable: the pipeline, vocabulary, and task stayed constant while the model gained the ability to reason over token order and context.
Phase 3: full DistilBERT fine-tuning
The final stage moved to DistilBERT and its WordPiece tokenizer. I unfroze the complete model and fine-tuned every parameter with a learning rate of 2 × 10^-5. Validation loss and accuracy were monitored together so that declining training loss could not be mistaken for continued generalization.
Epoch Train loss Validation accuracy Status 1 0.4609 83.06% — 2 0.3356 83.32% Peak 3 0.2602 82.86% Overfitting 5 0.1377 81.42% Stopped
Training results from the full-parameter DistilBERT fine-tuning run.
What changed in three years
The original project focused on finding an algorithm that worked. The rebuilt project focused on constructing evidence: initialization checks, controlled overfitting, explicit tensor shapes, staged architectural changes, and early stopping based on validation behavior.
The best model reached 83.32% validation accuracy at epoch two. The number matters, but the larger result is the transition from assembling a text-classification script to engineering and validating an end-to-end learning system.