# DendroNet: MNIST Classification with Dendritic Optimization

## Intro

### Project Overview

This hackathon submission demonstrates the power of **dendritic optimization** applied to MNIST handwritten digit classification. MNIST is a standard benchmark dataset used to validate new neural network architectures and optimization techniques. This project integrates **Perforated AI's dendritic layers** into a PyTorch CNN to achieve improved classification accuracy through adaptive neural network restructuring.

### Team

**Shreyas** - CS Student
Linkedin: https://www.linkedin.com/in/shreyas-nikam-a474b2361/

---

## Project Impact

MNIST serves as the foundation for Optical Character Recognition (OCR) systems that power document digitization, automated data entry, and text extraction from images. Improving OCR accuracy isn't just about better benchmarks—it has real-world implications:

- **Financial Services**: Automated check processing, invoice scanning, and tax document analysis require near-perfect accuracy to avoid costly errors in monetary transactions
- **Healthcare**: Medical record digitization and prescription reading systems where misread characters could impact patient care
- **Legal Industry**: Contract analysis and document management where accuracy is critical for compliance and liability
- **Accessibility**: Screen readers and assistive technologies that help visually impaired users access printed materials

By demonstrating that dendritic optimization can reduce classification errors by over 20%, this project shows how biologically-inspired neural architectures can significantly improve mission-critical OCR applications where every percentage point matters.

---

## Usage Instructions

### Installation

1. Clone this repository and navigate to the project folder:
```bash
git clone <repository-url>
cd DendroNet
```

2. Install the required dependencies:
```bash
pip install -r requirements.txt
```

3. *(Optional)* Request a Perforated AI license for access to advanced dendritic features:  
[https://share.hsforms.com/1meVnVMXtQTiUDm6WWKLGDgrzzmm](https://share.hsforms.com/1meVnVMXtQTiUDm6WWKLGDgrzzmm)

### Running the Project

#### Train the Dendritic MNIST Model

To train the model with dendritic optimization and Weights & Biases logging:

```bash
PAIPASSWORD=123 CUDA_VISIBLE_DEVICES=0 python mnist_perforatedai_wandb.py --count 25
```

**Key Parameters:**
- `PAIPASSWORD`: License password for Perforated AI library (use `123` for demo)
- `CUDA_VISIBLE_DEVICES`: GPU device ID (`0` for first GPU, or omit for CPU)
- `--count`: Number of hyperparameter sweep runs (default: 300)

**Additional Options:**
```bash
# Use specific hyperparameters without wandb
python mnist_perforatedai_wandb.py --use-config-string "Dendrites-2_0.25_0.0001_1_0.01_0_0"

# Adjust network width
python mnist_perforatedai_wandb.py --width 1.5

# Change learning rate
python mnist_perforatedai_wandb.py --lr 0.8 --gamma 0.65
```

#### Train the Baseline Model (No Dendrites)

To compare against the original architecture without dendritic optimization:

```bash
python mnist_original.py
```

This uses the standard PyTorch MNIST example with the same CNN architecture.

---

## Results

This implementation demonstrates that **Dendritic Optimization significantly outperforms traditional training**. Comparing the best traditional model to the best dendritic model:

| Model | Final Test Accuracy | Architecture | Parameter Count |
|-------|---------------------|--------------|-----------------|
| Traditional (Baseline) | 99.26% | Standard CNN | ~1.2M parameters |
| Dendritic (with Perforated AI) | 99.42% | CNN + Dendrites | ~1.2M + dendrites |

### Key Performance Metrics

- **Absolute Accuracy Gain**: +0.16 percentage points
- **Remaining Error Reduction**: **21.6%**
- **Training Methodology**: Automated hyperparameter search with Perforated Backpropagation (PB)

#### Understanding Remaining Error Reduction

The **Remaining Error Reduction (RER)** metric quantifies what percentage of the original model's errors were eliminated by dendrites:

```
Original Model Error: 100% - 99.26% = 0.74%
Dendritic Model Error: 100% - 99.42% = 0.58%
Error Eliminated: 0.74% - 0.58% = 0.16 percentage points
Remaining Error Reduction = (0.16 / 0.74) × 100 = 21.6%
```

**This means dendrites eliminated over one-fifth (21.6%) of the classification errors**, demonstrating the effectiveness of adaptive neural restructuring.

---

## Raw Results Graph (Required)

The following graph is **automatically generated** by the Perforated AI library during training. This is the **mandatory verification** showing that dendrites were properly integrated and optimized:

![Perforated AI Training Results](./PAI.png)

### Graph Interpretation

**Top-Left Panel (Main Results)**:
- **Blue line**: Validation accuracy (fluctuating test performance during training)
- **Orange line**: Validation running average (smoothed trend)
- **Green line**: Training accuracy
- **Blue dot**: Global best validation score achieved
- **Green dot**: Epoch where last improvement occurred
- **Red vertical lines**: CC dendrite additions (visible with PAI license)

The graph clearly demonstrates:
1. ✅ Progressive improvement over ~300 epochs
2. ✅ Dendritic network achieving significantly higher scores than baseline
3. ✅ Successful convergence with optimal performance marked
4. ✅ Multiple dendrite additions improving the architecture

**Other Panels**:
- **Top-Right**: Best personal record scores across evolution steps
- **Bottom-Left**: Adaptive learning rate scheduling
- **Bottom-Right**: Normalized fitness metrics tracking optimization progress

> [!IMPORTANT]
> This automatically-generated graph is **required for submission validation**. It proves that dendrites were properly integrated and that the Perforated AI library was correctly configured.

---

## Clean Results Graph (Optional)

For easier visualization, here's a condensed comparison highlighting the accuracy improvement:

![MNIST Accuracy Improvement](./accuracy_improvement.png)

This bar chart provides a clear visual representation of the performance gain achieved through dendritic optimization.

---

## Detailed Training Metrics (Optional)

The Perforated AI library generates comprehensive training visualizations:

![Detailed PAI Metrics](./PAI_detailed.png)

**Panel Breakdown**:
- **Top-Left**: Score progression showing validation, running average, and training accuracy over epochs
- **Top-Right**: Personal best scores tracked across different dendrite configurations
- **Bottom-Left**: Adaptive learning rate schedule with periodic resets during restructuring
- **Bottom-Right**: Fitness evolution normalized by training iterations

These metrics illustrate:
- How dendritic optimization explored the neural architecture space
- The automatic learning rate adaptation during network restructuring
- The continuous improvement in model fitness throughout training

---

## Technical Implementation

### Architecture Details

**Base Model**: Convolutional Neural Network (CNN)
```python
class Net(nn.Module):
    - Conv2d(1 → 32, kernel=3)
    - Conv2d(32 → 64, kernel=3)
    - MaxPool2d(kernel=2)
    - Dropout(p=0.25)
    - Linear(9216 → 128)
    - Dropout(p=0.5)
    - Linear(128 → 10)
```

**Optimization Method**: Perforated Backpropagation (PB) with CC Dendrites
- **Dendritic Mode**: 2 (CC dendrites with Perforated Backpropagation)
- **Max Dendrites**: 5 per restructuring event
- **Improvement Threshold**: Adaptive `[0.001, 0.0001, 0]`
- **Forward Function**: Sigmoid activation for dendrites
- **Weight Initialization**: 0.01× multiplier for candidate dendrites

**Framework**: PyTorch 2.0+  
**Logging**: Weights & Biases for experiment tracking  
**Optimizer**: Adadelta (lr=1.0, γ=0.7)

### Key Code Features

1. **Automated Dendritic Addition**: The Perforated AI tracker automatically adds dendrites when validation accuracy plateaus
2. **Architecture Restructuring**: Optimizer and learning rate scheduler automatically reset after dendrite insertion
3. **Hyperparameter Sweep**: Wandb integration for exploring dropout, weight decay, and dendritic configurations
4. **Dual Dataset Support**: Code supports both MNIST and EMNIST datasets

### File Structure

```
DendroNet/
├── README.md                          # This comprehensive guide
├── requirements.txt                   # Python dependencies
├── mnist_original.py                  # Baseline model (no dendrites)
├── mnist_dendritic_modified.py        # Modified dendritic implementation
├── mnist_perforatedai_wandb.py        # Main training script with W&B integration
├── PAI.png                            # Raw results graph (REQUIRED)
├── accuracy_improvement.png           # Clean comparison visualization
├── PAI_detailed.png                   # Detailed training metrics
└── data/                              # MNIST dataset (auto-downloaded)
```

---

## Dependencies

See [`requirements.txt`](./requirements.txt) for the complete dependency list:

```txt
torch>=2.0.0           # PyTorch deep learning framework
torchvision>=0.15.0    # MNIST dataset and image utilities
perforatedai           # Dendritic optimization library
wandb                  # Experiment tracking and visualization
argparse               # Command-line argument parsing
```

---

## Hyperparameter Search Space

The Weights & Biases sweep explores the following configuration space:

| Parameter | Values Tested | Description |
|-----------|---------------|-------------|
| `dropout` | [0.1, 0.25, 0.3, 0.5] | Dropout probability for regularization |
| `weight_decay` | [0, 0.0001, 0.001] | L2 regularization strength |
| `improvement_threshold` | [0, 1, 2] | Dendrite addition sensitivity |
| `candidate_weight_initialization_multiplier` | [0.01, 0.1] | Initial dendrite weight scale |
| `pai_forward_function` | [0=sigmoid, 1=relu, 2=tanh] | Dendritic activation function |
| `dendrite_mode` | [0=none, 1=GD, 2=CC/PB] | Dendritic optimization mode |
| `dendrite_graph_mode` | [0, 1] | Graph-based dendrite routing |

**Best Configuration Found**:
- Dropout: 0.25
- Weight Decay: 0.0001
- Dendrite Mode: 2 (CC with Perforated Backpropagation)
- Forward Function: Sigmoid

---

## Verification & Reproducibility

### How to Verify This Submission

1. **Check the Raw Results Graph** (`PAI.png`):
   - Verify the top-left panel shows progressive improvement
   - Confirm validation scores reach ~99.4% or higher
   - Look for the characteristic dendritic training pattern

2. **Run the Code Yourself**:
```bash
pip install -r requirements.txt
PAIPASSWORD=123 python mnist_perforatedai_wandb.py --count 5
```

3. **Compare with Baseline**:
```bash
python mnist_original.py --epochs 14
```
   Expected baseline: ~99.1-99.3% accuracy after 14 epochs

### Expected Training Time

- **Baseline Model**: ~5 minutes on GPU, ~15 minutes on CPU
- **Dendritic Model**: ~2-4 hours for full sweep with 25 configurations
- **Single Run**: ~5-10 minutes per configuration

---

## Future Improvements

Potential directions for extending this work:

### Immediate Next Steps
- Explore EMNIST (47 classes) to test dendritic scaling
- Implement data augmentation (rotation, scaling) for robustness
- Test different network widths and depths

### Advanced Exploration
- Apply to CIFAR-10/100 for color image classification
- Test on larger architectures (ResNet, Vision Transformers)
- Investigate dendritic pruning for model compression
- Benchmark computational cost vs. accuracy trade-offs
- Explore transfer learning with pre-trained dendritic networks

---

## Acknowledgments

- **Perforated AI** for the dendritic optimization library and hosting this hackathon
- **PyTorch Team** for the exceptional deep learning framework
- **Weights & Biases** for powerful experiment tracking tools
- **Yann LeCun et al.** for the original MNIST dataset

---

## License & Citation

This project is submitted as part of the **Perforated AI Hackathon** (2026).

If you use this code or build upon this work, please cite:
```
DendroNet: MNIST Classification with Dendritic Optimization
Vijay, 2026
Perforated AI Hackathon Submission
```

---

## Contact & Support

For questions about this submission:
- Open an issue in this repository
- Reach out via [your contact method]

For Perforated AI library support:
- Documentation: [Perforated AI Docs]
- License Request: [https://share.hsforms.com/1meVnVMXtQTiUDm6WWKLGDgrzzmm](https://share.hsforms.com/1meVnVMXtQTiUDm6WWKLGDgrzzmm)