--- license: apache-2.0 language: - en metrics: - accuracy base_model: - microsoft/resnet-18 tags: - perforated - perforated-ai - dendritic - dendritic-optimization - artificial-dendrite-network - imagenet - resnet - dendrite - artificial-dendrites - 12.3m model-index: - name: resnet-18-perforated results: - task: type: image-classification name: Image Classification dataset: name: ImageNet-1k type: imagenet-1k metrics: - type: accuracy value: 71.90 name: Top-1 Accuracy verified: false - type: parameter-count value: 13.3 name: Parameters (M) - task: type: image-classification name: Image Classification (Transfer Learning) dataset: name: Flowers-102 type: flowers-102 metrics: - type: accuracy value: 91.167 name: Top-1 Accuracy verified: false - type: parameter-count value: 12.3 name: Parameters (M) --- # ResNet-18 Perforated via Gradient Descent ## Model Description This is a **perforated** ResNet-18 model, one enhanced with **dendritic optimization**, a novel technique that increases accuracy with significantly improved parameter efficiency. The model was pretrained on ImageNet and can be used achieve for transfer learning with closer to ResNet-34 level performance while using a fraction of the parameters. There are two models in this family. [resnet-18-perforated-gd](https://huggingface.co/perforated-ai/resnet-18-perforated-gd) has 5 dendrites which were trained with gradient descent. [resnet-18-perforated-cascor](https://huggingface.co/perforated-ai/resnet-18-perforated-cascor) has 2 dendrites added and trained dendrites with Perforated Backpropagation via the Cascade Correlation learning paradigm. Experimental results can be seen in the spreadsheet [here](https://docs.google.com/spreadsheets/d/11Ux_aYvoRzOjr7lYHewc5LGybW8Bz_rSfqYK30tCbMU/edit?usp=sharing). ### What is Dendritic Optimization? Dendritic optimization (referred to as "perforation") enhances neural networks by adding specialized **dendrite nodes** to individual neurons. Each dendrite node receives the same inputs as its parent neuron but provides a dedicated output channel exclusively to that neuron. This allows neurons to make more sophisticated decisions about the features they encode, similar to how biological dendrites help neurons process information. Think of it as giving each neuron its own set of specialized advisors that help it better understand the features it's responsible for detecting. ## Model Details - **Architecture**: ResNet-18 with perforated pre-FC layer - **Parameters**: 12.3M (compared to 11.7M for standard ResNet-18, 21.8M for ResNet-34) - **Dendrites**: 5 dendrite nodes per neuron in the pre-FC layer - **Pre-training**: ImageNet-1k - **Framework**: PyTorch - **License**: Apache 2.0 ### Architecture Flow ``` Main ResNet-18 backbone Pre-FC Layer (Perforated FC layer with 3 dendrites) FC Layer (replaceable for transfer learning) ``` The uploaded FC layer has randomized weights, this model is only for use in transfer learning where the FC layer should be replaced. ## Performance ### Parameter Efficiency on ImageNet The key advantage of dendritic optimization is dramatic improvement in parameter efficiency. When considering the trade-off of moving from ResNet-18 to ResNet-34: | Model | Parameters | ImageNet Accuracy | Percentage Gain per M Params | |-------|-----------|-------------------|-------------------| | ResNet-18 | 11.7M | 69.76% | - (baseline) | | ResNet-34 | 21.8M | 73.30% | **0.35** | | ResNet-18-prefc-gd + 1 Dendrites | 12.2M | 70.99% | **2.34** | | ResNet-18-prefc-gd + 2 Dendrites | 12.5M | 71.37% | **2.03** | | ResNet-18-prefc-gd + 3 Dendrites | 12.7M | 71.65% | **1.79** | | ResNet-18-prefc-gd + 4 Dendrites | 13.0M | 71.83% | **1.57** | | ResNet-18-prefc-gd + 5 Dendrites | 13.3M | 71.90% | **1.34** | | ResNet-18-prefc-cascor + 1 Dendrites | 12.2M | 71.33% | **2.98** | | ResNet-18-prefc-cascor + 2 Dendrites | 12.5M | 71.73% | **2.49** | **Key Insight**: Adding dendrites to ResNet-18 provides 4-7x better accuracy improvement per additional parameter compared to upgrading to ResNet-34. These numbers are acheived without the pre-fc layer where the output FC layer is perforated. That model can be uploaded as well if requested. ### Transfer Learning Performance This model excels at transfer learning tasks: | Model | Parameters | Flowers-102 Accuracy |Oxford Pets Accuracy |Food-101 Accuracy | |-------|------------|----------------------|---------------------|------------------| | ResNet-18 | 11.2M | 89.8% | 90.8% | 81.7% | | ResNet-34 | 21.3M | 90.7% | 92.6% | 83.9% | | ResNet-18 Perforated (this model) | 12.5M | 91.2% | 91.8% | 82.4% | **Result**: For Pets and Food-101 this model closes 57% and 30%, respecively, of the accuracy gap with only 13% of the parameter gap between ResNet-18 and ResNet-34. While on the Flowers-102 dataset the perforated resnet-18 outperforms the resnet-34 with over 2x the parameter effeciency. Full results of all experiments performed can be found [here](https://docs.google.com/spreadsheets/d/11Ux_aYvoRzOjr7lYHewc5LGybW8Bz_rSfqYK30tCbMU/edit?usp=sharing). ### Latency The original model started from the torchvision ResNet rather than the microsoft/resnet. Latency comparisons start there as well to ensure fair results. Test was run processing a single image at a time on a AMD Ryzen Threadripper PRO 9975WX CPU over the full Flowers-102 dataset. | Model | Time Per Image | Throughput | |-------|-----------|---------------------| | ResNet-18 | 4.04ms | 247.46 FPS | | ResNet-34 | 7.48ms | 133.74 FPS | | ResNet-18 Perforated | 4.37ms | 228.63 FPS | As expected, latency is proportinally similar to parameter count between the models. ## Usage ### Installation First, install the PerforatedAI library: ```bash pip install perforatedai ``` ### Loading the Model ```python import torch import torchvision from perforatedai import utils_perforatedai as UPA from perforatedai import library_perforatedai as LPA # Create base model architecture base_model = torchvision.models.get_model('resnet18', weights=None, num_classes=1000) # Convert to perforated architecture model = LPA.ResNetPAIPreFC(base_model) # Load pretrained weights from HuggingFace model = UPA.from_hf_pretrained(model, 'perforated-ai/resnet-18-perforated-gd') ``` ### Using for Transfer Learning For transfer learning on your own dataset, replace the final FC layer: ```python import torch.nn as nn # Assuming you have num_classes for your task num_classes = 102 # Example: Flowers-102 # Replace the final FC layer model.fc = nn.Linear(model.fc.in_features, num_classes) ``` ### Preprocessing Use standard preprocessing that you would use with any ResNet-18 such as the following for ImageNet: ```python from torchvision import transforms transform = transforms.Compose([ transforms.Resize(256), transforms.CenterCrop(224), transforms.ToTensor(), transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]) ]) ``` ## Intended Use ### Primary Use Cases - **Transfer learning** for image classification tasks - **Resource-constrained environments** where parameter efficiency is critical - **Applications requiring ResNet-34 performance** with ResNet-18 computational costs ### Recommendations This model is particularly effective when you need strong feature representations without the parameter overhead of larger models. The perforated pre-FC layer has learned rich representations from ImageNet that transfer well to downstream tasks. ## Training Details ### Training Data - **Dataset**: ImageNet-1k (1.28M training images, 1000 classes) - **Perforation Target**: Pre-FC layer (for transfer learning optimization) ### Training Procedure Detailed training code and procedures are available in our GitHub repository: - **Main Repository**: [link](https://github.com/PerforatedAI/PerforatedAI) - **Training Scripts**: [link](https://github.com/PerforatedAI/PerforatedAI/tree/main/Examples/imagenet) ## Evaluation ### Evaluation Dataset The model was evaluated on Flowers-102 dataset for transfer learning performance. **Evaluation Code**: [link](https://github.com/PerforatedAI/PerforatedAI/tree/main/Examples/imagenet_pretrained) ## Citation If you use this model in your research, please cite: ```bibtex @article{perforatedai2025, title={Perforated Backpropagation: A Neuroscience Inspired Extension to Artificial Neural Networks}, author={[Rorry Brenner, and Laurent Itti]}, journal={arXiv preprint arXiv:2501.18018}, year={2025}, url={https://arxiv.org/abs/2501.18018} } ``` ## Additional Resources - **Paper**: https://arxiv.org/abs/2501.18018 - **GitHub**: https://github.com/PerforatedAI/PerforatedAI - **PyPI Package**: https://pypi.org/project/perforatedai/ ## Model Card Authors PerforatedAI Team ## Model Card Contact For questions or issues, please open an issue on our [GitHub repository](https://github.com/PerforatedAI/PerforatedAI).