diff --git a/README.md b/README.md index f7e4124..a0aa4c4 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,5 @@

- Goldeneye + GoldenBox

@@ -7,71 +7,36 @@

- Background • - Usage • + Overview • + Installing • Code • Acknowledgements • - Citation • + License

-## Background +## Overview -GoldenEye is a functional simulator with fault injection capabilities for common and emerging numerical formats, implemented for the PyTorch deep learning framework. GoldenEye provides a unified framework for numerical format evaluation of DNNs, including traditional number systems such as fixed and floating point, as well as recent DNN-inspired formats such as block floating point and AdaptivFloat. Additionally, GoldenEye enables single- and multi- bit flips at various logical and functional points during a value’s lifetime for resiliency analysis, including for the first time attention to numerical values’ hardware metadata. GoldenEye is an easy-to-use, extensible, versatile, and fast tool for dependability research and future DNN accelerator design. - -![](https://user-images.githubusercontent.com/89948656/176387208-cfd64047-3841-4abf-bf54-5d2e63f5a2e5.png) - - -## Usage - -Take a look at our documentation [here](https://goldeneyedocs.readthedocs.io/en/stable/index.html). +GoldenEye Object Detection is an extension of the original [GoldenEye](https://github.com//ma3mool/goldeneye) functional simulator with fault injection capabilities for common and emerging numerical formats. Previously the simulator was only supported image classification models. Now, we have extended our use case to object detection models as well. ### Installing -**Ubuntu with Sudo Privileges** -1. Recursively clone the goldeneye repository. +**Ubuntu-20.04 or later** +1. Clone the goldeneye repository. ```bash -git clone --recurse-submodules git@github.com:ma3mool/goldeneye.git +git clone https://github.com/sajidahmed12/goldeneye-object-detection ``` -2. Download ninja-build which is needed for qtorch. +2. Download ninja-build, which is needed for qtorch. ```bash sudo apt install ninja-build ``` -3. Download the other project dependencies. Please make sure you are inside the goldeneye folder when applying this command. +3. Install the other project dependencies from the requirements.txt file. ```bash pip install -r requirements.txt ``` -4. Setup environment variable (replace with the directory where the imagenet dataset is downloaded). -```bash -ML_DATASETS=/dir/to/imagenet/ -``` - -**Docker** -1. Recursively clone the goldeneye repository. -```bash -git clone --recurse-submodules git@github.com:ma3mool/goldeneye.git -``` - -2. Pull the goldeneye docker image and rename it to simply the next steps -```bash -docker pull goldeneyetool/goldeneye:latest -docker image tag goldeneyetool/goldeneye goldeneye -``` - -3. Within the goldeneye folder, run the shell on the pulled docker image. Make sure to replace [/path/to/imagenet] with the actual path to your downloaded imagenet dataset. -```bash -cd goldeneye -docker run -ti - --mount type=bind,source=`pwd`/src/,target=/src - --mount type=bind,source=`pwd`/val/,target=/val - --mount type=bind,source=`pwd`/scripts/,target=/scripts - --mount type=bind,source=[/path/to/imagenet],target=/datasets/imagenet - goldeneye -``` - ### Testing ```bash @@ -81,30 +46,42 @@ pytest val/test_num_sys.py ## Code ### Structure -The ```scripts``` folder includes wrappers around the goldeneye framework to simplify its use. The ```src``` folder contains all of the goldeneye core logic such as number system implementation and error injection routines. The ```val``` folder is used for unit-testing the code. You can run it using pytest to check that the installation process was successful. +The ```scripts``` folder includes wrappers around the goldeneye-obj framework to simplify its use. The ```src``` folder contains all of the core components, such as number system implementation, error injection routines, dataloaders, etc. The ```val``` folder is used for unit testing the code. You can run it using pytest to check that the installation process was successful. -## Acknowledgements -- Tarek Aloui (Harvard) -- David Brooks (Harvard) -- Abdulrahman Mahmoud (Harvard) -- Joshua Park (Harvard) -- Thierry Tambe (Harvard) -- Gu-Yeon Wei (Harvard) -## Citation +Example Outputs are saved in this Google Drive [link](https://drive.google.com/drive/folders/1nP0pavu3vprPc9EvuahkF9UsETDlwFEp?usp=sharing) -If you use or reference Goldeneye, please cite: +## Example Commands +Pre-processing ``` -@INPROCEEDINGS{GoldeneyeMahmoudTambeDSN2022, -author={A. {Mahmoud} and T. {Tambe} and T. {Aloui} and D. {Brooks} and G. {Wei}}, -booktitle={2022 52nd Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN)}, -title={GoldenEye: A Platform for Evaluating Emerging Data Formats in DNN Accelerators}, -year={2022}, -} +python preprocess.py -b 16 -n frcnn -d COCO -w 8 -P FP32 -f fp_n -C 0 [MS-COCO FP32](MS-COCO FP32]) ``` -## License +Profiling +``` +python profiling.py -b 16 -n frcnn -d COCO -w 16 -P FP32 -f fp_n -B 32 -R 23 +``` + +Split Data +``` +python split_data.py -b 16 -n frcnn -d COCO -o -w 16 -P FP32 -f fp_n -B 32 -R 23 +``` + +Error Injections +``` +python injections.py -b 16 -n frcnn -d COCO -w 16 -P FP32 -i 102400 -I 1 -f fp_n -B 32 -R 23 +``` + +Post-processing +``` +python postprocess.py -b 16 -n frcnn -d COCO -w 16 -P FP32 -i 102400 -I 1 -f fp_n -B 32 -R 23 +``` +## Acknowledgements + +- This Repository was forked from [Goldeneye](https://github.com/ma3mool/goldeneye/) developed and maintained by [Sajid Ahmed](https://sajidahmed12.github.io) & [Dr. Abdulrahman Mahmoud](https://ma3mool.github.io/) + +## License MIT License diff --git a/data/put-your-data-here.txt b/data/put-your-data-here.txt new file mode 100644 index 0000000..e69de29 diff --git a/models/__init__.py b/models/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/models/densenet.py b/models/densenet.py new file mode 100644 index 0000000..ceb987d --- /dev/null +++ b/models/densenet.py @@ -0,0 +1,229 @@ +import os +from collections import OrderedDict + +import torch +import torch.nn as nn +import torch.nn.functional as F + +__all__ = ["DenseNet", "densenet121", "densenet169", "densenet161"] + + +class _DenseLayer(nn.Sequential): + def __init__(self, num_input_features, growth_rate, bn_size, drop_rate): + super(_DenseLayer, self).__init__() + self.add_module("norm1", nn.BatchNorm2d(num_input_features)), + self.add_module("relu1", nn.ReLU(inplace=True)), + self.add_module( + "conv1", + nn.Conv2d( + num_input_features, + bn_size * growth_rate, + kernel_size=1, + stride=1, + bias=False, + ), + ), + self.add_module("norm2", nn.BatchNorm2d(bn_size * growth_rate)), + self.add_module("relu2", nn.ReLU(inplace=True)), + self.add_module( + "conv2", + nn.Conv2d( + bn_size * growth_rate, + growth_rate, + kernel_size=3, + stride=1, + padding=1, + bias=False, + ), + ), + self.drop_rate = drop_rate + + def forward(self, x): + new_features = super(_DenseLayer, self).forward(x) + if self.drop_rate > 0: + new_features = F.dropout( + new_features, p=self.drop_rate, training=self.training + ) + return torch.cat([x, new_features], 1) + + +class _DenseBlock(nn.Sequential): + def __init__(self, num_layers, num_input_features, bn_size, growth_rate, drop_rate): + super(_DenseBlock, self).__init__() + for i in range(num_layers): + layer = _DenseLayer( + num_input_features + i * growth_rate, growth_rate, bn_size, drop_rate + ) + self.add_module("denselayer%d" % (i + 1), layer) + + +class _Transition(nn.Sequential): + def __init__(self, num_input_features, num_output_features): + super(_Transition, self).__init__() + self.add_module("norm", nn.BatchNorm2d(num_input_features)) + self.add_module("relu", nn.ReLU(inplace=True)) + self.add_module( + "conv", + nn.Conv2d( + num_input_features, + num_output_features, + kernel_size=1, + stride=1, + bias=False, + ), + ) + self.add_module("pool", nn.AvgPool2d(kernel_size=2, stride=2)) + + +class DenseNet(nn.Module): + r"""Densenet-BC model class, based on + `"Densely Connected Convolutional Networks" `_ + + Args: + growth_rate (int) - how many filters to add each layer (`k` in paper) + block_config (list of 4 ints) - how many layers in each pooling block + num_init_features (int) - the number of filters to learn in the first convolution layer + bn_size (int) - multiplicative factor for number of bottle neck layers + (i.e. bn_size * k features in the bottleneck layer) + drop_rate (float) - dropout rate after each dense layer + num_classes (int) - number of classification classes + """ + + def __init__( + self, + growth_rate=32, + block_config=(6, 12, 24, 16), + num_init_features=64, + bn_size=4, + drop_rate=0, + num_classes=10, + ): + + super(DenseNet, self).__init__() + + # First convolution + + # CIFAR-10: kernel_size 7 ->3, stride 2->1, padding 3->1 + self.features = nn.Sequential( + OrderedDict( + [ + ( + "conv0", + nn.Conv2d( + 3, + num_init_features, + kernel_size=3, + stride=1, + padding=1, + bias=False, + ), + ), + ("norm0", nn.BatchNorm2d(num_init_features)), + ("relu0", nn.ReLU(inplace=True)), + ("pool0", nn.MaxPool2d(kernel_size=3, stride=2, padding=1)), + ] + ) + ) + # END + + # Each denseblock + num_features = num_init_features + for i, num_layers in enumerate(block_config): + block = _DenseBlock( + num_layers=num_layers, + num_input_features=num_features, + bn_size=bn_size, + growth_rate=growth_rate, + drop_rate=drop_rate, + ) + self.features.add_module("denseblock%d" % (i + 1), block) + num_features = num_features + num_layers * growth_rate + if i != len(block_config) - 1: + trans = _Transition( + num_input_features=num_features, + num_output_features=num_features // 2, + ) + self.features.add_module("transition%d" % (i + 1), trans) + num_features = num_features // 2 + + # Final batch norm + self.features.add_module("norm5", nn.BatchNorm2d(num_features)) + + # Linear layer + self.classifier = nn.Linear(num_features, num_classes) + + # Official init from torch repo. + for m in self.modules(): + if isinstance(m, nn.Conv2d): + nn.init.kaiming_normal_(m.weight) + elif isinstance(m, nn.BatchNorm2d): + nn.init.constant_(m.weight, 1) + nn.init.constant_(m.bias, 0) + elif isinstance(m, nn.Linear): + nn.init.constant_(m.bias, 0) + + def forward(self, x): + features = self.features(x) + out = F.relu(features, inplace=True) + out = F.adaptive_avg_pool2d(out, (1, 1)).view(features.size(0), -1) + out = self.classifier(out) + return out + + +def _densenet( + arch, + growth_rate, + block_config, + num_init_features, + pretrained, + progress, + device, + **kwargs +): + model = DenseNet(growth_rate, block_config, num_init_features, **kwargs) + if pretrained: + script_dir = os.path.dirname(__file__) + state_dict = torch.load( + script_dir + "/state_dicts/" + arch + ".pt", map_location=device + ) + model.load_state_dict(state_dict) + return model + + +def densenet121(pretrained=False, progress=True, device="cpu", **kwargs): + r"""Densenet-121 model from + `"Densely Connected Convolutional Networks" `_ + + Args: + pretrained (bool): If True, returns a model pre-trained on ImageNet + progress (bool): If True, displays a progress bar of the download to stderr + """ + return _densenet( + "densenet121", 32, (6, 12, 24, 16), 64, pretrained, progress, device, **kwargs + ) + + +def densenet161(pretrained=False, progress=True, device="cpu", **kwargs): + r"""Densenet-161 model from + `"Densely Connected Convolutional Networks" `_ + + Args: + pretrained (bool): If True, returns a model pre-trained on ImageNet + progress (bool): If True, displays a progress bar of the download to stderr + """ + return _densenet( + "densenet161", 48, (6, 12, 36, 24), 96, pretrained, progress, device, **kwargs + ) + + +def densenet169(pretrained=False, progress=True, device="cpu", **kwargs): + r"""Densenet-169 model from + `"Densely Connected Convolutional Networks" `_ + + Args: + pretrained (bool): If True, returns a model pre-trained on ImageNet + progress (bool): If True, displays a progress bar of the download to stderr + """ + return _densenet( + "densenet169", 32, (6, 12, 32, 32), 64, pretrained, progress, device, **kwargs + ) diff --git a/models/googlenet.py b/models/googlenet.py new file mode 100644 index 0000000..afaa8e9 --- /dev/null +++ b/models/googlenet.py @@ -0,0 +1,227 @@ +import os +from collections import namedtuple + +import torch +import torch.nn as nn +import torch.nn.functional as F + +__all__ = ["GoogLeNet", "googlenet"] + + +_GoogLeNetOuputs = namedtuple( + "GoogLeNetOuputs", ["logits", "aux_logits2", "aux_logits1"] +) + + +def googlenet(pretrained=False, progress=True, device="cpu", **kwargs): + r"""GoogLeNet (Inception v1) model architecture from + `"Going Deeper with Convolutions" `_. + + Args: + pretrained (bool): If True, returns a model pre-trained on ImageNet + progress (bool): If True, displays a progress bar of the download to stderr + aux_logits (bool): If True, adds two auxiliary branches that can improve training. + Default: *False* when pretrained is True otherwise *True* + transform_input (bool): If True, preprocesses the input according to the method with which it + was trained on ImageNet. Default: *False* + """ + model = GoogLeNet() + if pretrained: + script_dir = os.path.dirname(__file__) + state_dict = torch.load( + script_dir + "/state_dicts/googlenet.pt", map_location=device + ) + model.load_state_dict(state_dict) + return model + + +class GoogLeNet(nn.Module): + + # CIFAR10: aux_logits True->False + def __init__(self, num_classes=10, aux_logits=False, transform_input=False): + super(GoogLeNet, self).__init__() + self.aux_logits = aux_logits + self.transform_input = transform_input + + # CIFAR10: out_channels 64->192, kernel_size 7->3, stride 2->1, padding 3->1 + self.conv1 = BasicConv2d(3, 192, kernel_size=3, stride=1, padding=1) + # self.maxpool1 = nn.MaxPool2d(3, stride=2, ceil_mode=True) + # self.conv2 = BasicConv2d(64, 64, kernel_size=1) + # self.conv3 = BasicConv2d(64, 192, kernel_size=3, padding=1) + # self.maxpool2 = nn.MaxPool2d(3, stride=2, ceil_mode=True) + # END + + self.inception3a = Inception(192, 64, 96, 128, 16, 32, 32) + self.inception3b = Inception(256, 128, 128, 192, 32, 96, 64) + + # CIFAR10: padding 0->1, ciel_model True->False + self.maxpool3 = nn.MaxPool2d(3, stride=2, padding=1, ceil_mode=False) + # END + + self.inception4a = Inception(480, 192, 96, 208, 16, 48, 64) + self.inception4b = Inception(512, 160, 112, 224, 24, 64, 64) + self.inception4c = Inception(512, 128, 128, 256, 24, 64, 64) + self.inception4d = Inception(512, 112, 144, 288, 32, 64, 64) + self.inception4e = Inception(528, 256, 160, 320, 32, 128, 128) + + # CIFAR10: kernel_size 2->3, padding 0->1, ciel_model True->False + self.maxpool4 = nn.MaxPool2d(3, stride=2, padding=1, ceil_mode=False) + # END + + self.inception5a = Inception(832, 256, 160, 320, 32, 128, 128) + self.inception5b = Inception(832, 384, 192, 384, 48, 128, 128) + + if aux_logits: + self.aux1 = InceptionAux(512, num_classes) + self.aux2 = InceptionAux(528, num_classes) + + self.avgpool = nn.AdaptiveAvgPool2d((1, 1)) + self.dropout = nn.Dropout(0.2) + self.fc = nn.Linear(1024, num_classes) + + # if init_weights: + # self._initialize_weights() + + # def _initialize_weights(self): + # for m in self.modules(): + # if isinstance(m, nn.Conv2d) or isinstance(m, nn.Linear): + # import scipy.stats as stats + # X = stats.truncnorm(-2, 2, scale=0.01) + # values = torch.as_tensor(X.rvs(m.weight.numel()), dtype=m.weight.dtype) + # values = values.view(m.weight.size()) + # with torch.no_grad(): + # m.weight.copy_(values) + # elif isinstance(m, nn.BatchNorm2d): + # nn.init.constant_(m.weight, 1) + # nn.init.constant_(m.bias, 0) + + def forward(self, x): + if self.transform_input: + x_ch0 = torch.unsqueeze(x[:, 0], 1) * (0.229 / 0.5) + (0.485 - 0.5) / 0.5 + x_ch1 = torch.unsqueeze(x[:, 1], 1) * (0.224 / 0.5) + (0.456 - 0.5) / 0.5 + x_ch2 = torch.unsqueeze(x[:, 2], 1) * (0.225 / 0.5) + (0.406 - 0.5) / 0.5 + x = torch.cat((x_ch0, x_ch1, x_ch2), 1) + + # N x 3 x 224 x 224 + x = self.conv1(x) + + # CIFAR10 + # N x 64 x 112 x 112 + # x = self.maxpool1(x) + # N x 64 x 56 x 56 + # x = self.conv2(x) + # N x 64 x 56 x 56 + # x = self.conv3(x) + # N x 192 x 56 x 56 + # x = self.maxpool2(x) + # END + + # N x 192 x 28 x 28 + x = self.inception3a(x) + # N x 256 x 28 x 28 + x = self.inception3b(x) + # N x 480 x 28 x 28 + x = self.maxpool3(x) + # N x 480 x 14 x 14 + x = self.inception4a(x) + # N x 512 x 14 x 14 + if self.training and self.aux_logits: + aux1 = self.aux1(x) + + x = self.inception4b(x) + # N x 512 x 14 x 14 + x = self.inception4c(x) + # N x 512 x 14 x 14 + x = self.inception4d(x) + # N x 528 x 14 x 14 + if self.training and self.aux_logits: + aux2 = self.aux2(x) + + x = self.inception4e(x) + # N x 832 x 14 x 14 + x = self.maxpool4(x) + # N x 832 x 7 x 7 + x = self.inception5a(x) + # N x 832 x 7 x 7 + x = self.inception5b(x) + # N x 1024 x 7 x 7 + + x = self.avgpool(x) + # N x 1024 x 1 x 1 + x = x.view(x.size(0), -1) + # N x 1024 + x = self.dropout(x) + x = self.fc(x) + # N x 1000 (num_classes) + if self.training and self.aux_logits: + return _GoogLeNetOuputs(x, aux2, aux1) + return x + + +class Inception(nn.Module): + def __init__(self, in_channels, ch1x1, ch3x3red, ch3x3, ch5x5red, ch5x5, pool_proj): + super(Inception, self).__init__() + + self.branch1 = BasicConv2d(in_channels, ch1x1, kernel_size=1) + + self.branch2 = nn.Sequential( + BasicConv2d(in_channels, ch3x3red, kernel_size=1), + BasicConv2d(ch3x3red, ch3x3, kernel_size=3, padding=1), + ) + + self.branch3 = nn.Sequential( + BasicConv2d(in_channels, ch5x5red, kernel_size=1), + BasicConv2d(ch5x5red, ch5x5, kernel_size=3, padding=1), + ) + + self.branch4 = nn.Sequential( + nn.MaxPool2d(kernel_size=3, stride=1, padding=1, ceil_mode=True), + BasicConv2d(in_channels, pool_proj, kernel_size=1), + ) + + def forward(self, x): + branch1 = self.branch1(x) + branch2 = self.branch2(x) + branch3 = self.branch3(x) + branch4 = self.branch4(x) + + outputs = [branch1, branch2, branch3, branch4] + return torch.cat(outputs, 1) + + +class InceptionAux(nn.Module): + def __init__(self, in_channels, num_classes): + super(InceptionAux, self).__init__() + self.conv = BasicConv2d(in_channels, 128, kernel_size=1) + + self.fc1 = nn.Linear(2048, 1024) + self.fc2 = nn.Linear(1024, num_classes) + + def forward(self, x): + # aux1: N x 512 x 14 x 14, aux2: N x 528 x 14 x 14 + x = F.adaptive_avg_pool2d(x, (4, 4)) + # aux1: N x 512 x 4 x 4, aux2: N x 528 x 4 x 4 + x = self.conv(x) + # N x 128 x 4 x 4 + x = x.view(x.size(0), -1) + # N x 2048 + x = F.relu(self.fc1(x), inplace=True) + # N x 2048 + x = F.dropout(x, 0.7, training=self.training) + # N x 2048 + x = self.fc2(x) + # N x 1024 + + return x + + +class BasicConv2d(nn.Module): + def __init__(self, in_channels, out_channels, **kwargs): + super(BasicConv2d, self).__init__() + self.conv = nn.Conv2d(in_channels, out_channels, bias=False, **kwargs) + self.bn = nn.BatchNorm2d(out_channels, eps=0.001) + + def forward(self, x): + x = self.conv(x) + x = self.bn(x) + return F.relu(x, inplace=True) diff --git a/models/inception.py b/models/inception.py new file mode 100644 index 0000000..6e0e167 --- /dev/null +++ b/models/inception.py @@ -0,0 +1,336 @@ +import os +from collections import namedtuple + +import torch +import torch.nn as nn +import torch.nn.functional as F + +__all__ = ["Inception3", "inception_v3"] + + +_InceptionOuputs = namedtuple("InceptionOuputs", ["logits", "aux_logits"]) + + +def inception_v3(pretrained=False, progress=True, device="cpu", **kwargs): + r"""Inception v3 model architecture from + `"Rethinking the Inception Architecture for Computer Vision" `_. + + .. note:: + **Important**: In contrast to the other models the inception_v3 expects tensors with a size of + N x 3 x 299 x 299, so ensure your images are sized accordingly. + + Args: + pretrained (bool): If True, returns a model pre-trained on ImageNet + progress (bool): If True, displays a progress bar of the download to stderr + aux_logits (bool): If True, add an auxiliary branch that can improve training. + Default: *True* + transform_input (bool): If True, preprocesses the input according to the method with which it + was trained on ImageNet. Default: *False* + """ + model = Inception3() + if pretrained: + script_dir = os.path.dirname(__file__) + state_dict = torch.load( + script_dir + "/state_dicts/inception_v3.pt", map_location=device + ) + model.load_state_dict(state_dict) + return model + + +class Inception3(nn.Module): + # CIFAR10: aux_logits True->False + def __init__(self, num_classes=10, aux_logits=False, transform_input=False): + super(Inception3, self).__init__() + self.aux_logits = aux_logits + self.transform_input = transform_input + + # CIFAR10: stride 2->1, padding 0 -> 1 + self.Conv2d_1a_3x3 = BasicConv2d(3, 192, kernel_size=3, stride=1, padding=1) + # self.Conv2d_2a_3x3 = BasicConv2d(32, 32, kernel_size=3) + # self.Conv2d_2b_3x3 = BasicConv2d(32, 64, kernel_size=3, padding=1) + # self.Conv2d_3b_1x1 = BasicConv2d(64, 80, kernel_size=1) + # self.Conv2d_4a_3x3 = BasicConv2d(80, 192, kernel_size=3) + self.Mixed_5b = InceptionA(192, pool_features=32) + self.Mixed_5c = InceptionA(256, pool_features=64) + self.Mixed_5d = InceptionA(288, pool_features=64) + self.Mixed_6a = InceptionB(288) + self.Mixed_6b = InceptionC(768, channels_7x7=128) + self.Mixed_6c = InceptionC(768, channels_7x7=160) + self.Mixed_6d = InceptionC(768, channels_7x7=160) + self.Mixed_6e = InceptionC(768, channels_7x7=192) + if aux_logits: + self.AuxLogits = InceptionAux(768, num_classes) + self.Mixed_7a = InceptionD(768) + self.Mixed_7b = InceptionE(1280) + self.Mixed_7c = InceptionE(2048) + self.fc = nn.Linear(2048, num_classes) + + # for m in self.modules(): + # if isinstance(m, nn.Conv2d) or isinstance(m, nn.Linear): + # import scipy.stats as stats + # stddev = m.stddev if hasattr(m, 'stddev') else 0.1 + # X = stats.truncnorm(-2, 2, scale=stddev) + # values = torch.as_tensor(X.rvs(m.weight.numel()), dtype=m.weight.dtype) + # values = values.view(m.weight.size()) + # with torch.no_grad(): + # m.weight.copy_(values) + # elif isinstance(m, nn.BatchNorm2d): + # nn.init.constant_(m.weight, 1) + # nn.init.constant_(m.bias, 0) + + def forward(self, x): + if self.transform_input: + x_ch0 = torch.unsqueeze(x[:, 0], 1) * (0.229 / 0.5) + (0.485 - 0.5) / 0.5 + x_ch1 = torch.unsqueeze(x[:, 1], 1) * (0.224 / 0.5) + (0.456 - 0.5) / 0.5 + x_ch2 = torch.unsqueeze(x[:, 2], 1) * (0.225 / 0.5) + (0.406 - 0.5) / 0.5 + x = torch.cat((x_ch0, x_ch1, x_ch2), 1) + # N x 3 x 299 x 299 + x = self.Conv2d_1a_3x3(x) + + # CIFAR10 + # N x 32 x 149 x 149 + # x = self.Conv2d_2a_3x3(x) + # N x 32 x 147 x 147 + # x = self.Conv2d_2b_3x3(x) + # N x 64 x 147 x 147 + # x = F.max_pool2d(x, kernel_size=3, stride=2) + # N x 64 x 73 x 73 + # x = self.Conv2d_3b_1x1(x) + # N x 80 x 73 x 73 + # x = self.Conv2d_4a_3x3(x) + # N x 192 x 71 x 71 + # x = F.max_pool2d(x, kernel_size=3, stride=2) + # N x 192 x 35 x 35 + x = self.Mixed_5b(x) + # N x 256 x 35 x 35 + x = self.Mixed_5c(x) + # N x 288 x 35 x 35 + x = self.Mixed_5d(x) + # N x 288 x 35 x 35 + x = self.Mixed_6a(x) + # N x 768 x 17 x 17 + x = self.Mixed_6b(x) + # N x 768 x 17 x 17 + x = self.Mixed_6c(x) + # N x 768 x 17 x 17 + x = self.Mixed_6d(x) + # N x 768 x 17 x 17 + x = self.Mixed_6e(x) + # N x 768 x 17 x 17 + if self.training and self.aux_logits: + aux = self.AuxLogits(x) + # N x 768 x 17 x 17 + x = self.Mixed_7a(x) + # N x 1280 x 8 x 8 + x = self.Mixed_7b(x) + # N x 2048 x 8 x 8 + x = self.Mixed_7c(x) + # N x 2048 x 8 x 8 + # Adaptive average pooling + x = F.adaptive_avg_pool2d(x, (1, 1)) + # N x 2048 x 1 x 1 + x = F.dropout(x, training=self.training) + # N x 2048 x 1 x 1 + x = x.view(x.size(0), -1) + # N x 2048 + x = self.fc(x) + # N x 1000 (num_classes) + if self.training and self.aux_logits: + return _InceptionOuputs(x, aux) + return x + + +class InceptionA(nn.Module): + def __init__(self, in_channels, pool_features): + super(InceptionA, self).__init__() + self.branch1x1 = BasicConv2d(in_channels, 64, kernel_size=1) + + self.branch5x5_1 = BasicConv2d(in_channels, 48, kernel_size=1) + self.branch5x5_2 = BasicConv2d(48, 64, kernel_size=5, padding=2) + + self.branch3x3dbl_1 = BasicConv2d(in_channels, 64, kernel_size=1) + self.branch3x3dbl_2 = BasicConv2d(64, 96, kernel_size=3, padding=1) + self.branch3x3dbl_3 = BasicConv2d(96, 96, kernel_size=3, padding=1) + + self.branch_pool = BasicConv2d(in_channels, pool_features, kernel_size=1) + + def forward(self, x): + branch1x1 = self.branch1x1(x) + + branch5x5 = self.branch5x5_1(x) + branch5x5 = self.branch5x5_2(branch5x5) + + branch3x3dbl = self.branch3x3dbl_1(x) + branch3x3dbl = self.branch3x3dbl_2(branch3x3dbl) + branch3x3dbl = self.branch3x3dbl_3(branch3x3dbl) + + branch_pool = F.avg_pool2d(x, kernel_size=3, stride=1, padding=1) + branch_pool = self.branch_pool(branch_pool) + + outputs = [branch1x1, branch5x5, branch3x3dbl, branch_pool] + return torch.cat(outputs, 1) + + +class InceptionB(nn.Module): + def __init__(self, in_channels): + super(InceptionB, self).__init__() + self.branch3x3 = BasicConv2d(in_channels, 384, kernel_size=3, stride=2) + + self.branch3x3dbl_1 = BasicConv2d(in_channels, 64, kernel_size=1) + self.branch3x3dbl_2 = BasicConv2d(64, 96, kernel_size=3, padding=1) + self.branch3x3dbl_3 = BasicConv2d(96, 96, kernel_size=3, stride=2) + + def forward(self, x): + branch3x3 = self.branch3x3(x) + + branch3x3dbl = self.branch3x3dbl_1(x) + branch3x3dbl = self.branch3x3dbl_2(branch3x3dbl) + branch3x3dbl = self.branch3x3dbl_3(branch3x3dbl) + + branch_pool = F.max_pool2d(x, kernel_size=3, stride=2) + + outputs = [branch3x3, branch3x3dbl, branch_pool] + return torch.cat(outputs, 1) + + +class InceptionC(nn.Module): + def __init__(self, in_channels, channels_7x7): + super(InceptionC, self).__init__() + self.branch1x1 = BasicConv2d(in_channels, 192, kernel_size=1) + + c7 = channels_7x7 + self.branch7x7_1 = BasicConv2d(in_channels, c7, kernel_size=1) + self.branch7x7_2 = BasicConv2d(c7, c7, kernel_size=(1, 7), padding=(0, 3)) + self.branch7x7_3 = BasicConv2d(c7, 192, kernel_size=(7, 1), padding=(3, 0)) + + self.branch7x7dbl_1 = BasicConv2d(in_channels, c7, kernel_size=1) + self.branch7x7dbl_2 = BasicConv2d(c7, c7, kernel_size=(7, 1), padding=(3, 0)) + self.branch7x7dbl_3 = BasicConv2d(c7, c7, kernel_size=(1, 7), padding=(0, 3)) + self.branch7x7dbl_4 = BasicConv2d(c7, c7, kernel_size=(7, 1), padding=(3, 0)) + self.branch7x7dbl_5 = BasicConv2d(c7, 192, kernel_size=(1, 7), padding=(0, 3)) + + self.branch_pool = BasicConv2d(in_channels, 192, kernel_size=1) + + def forward(self, x): + branch1x1 = self.branch1x1(x) + + branch7x7 = self.branch7x7_1(x) + branch7x7 = self.branch7x7_2(branch7x7) + branch7x7 = self.branch7x7_3(branch7x7) + + branch7x7dbl = self.branch7x7dbl_1(x) + branch7x7dbl = self.branch7x7dbl_2(branch7x7dbl) + branch7x7dbl = self.branch7x7dbl_3(branch7x7dbl) + branch7x7dbl = self.branch7x7dbl_4(branch7x7dbl) + branch7x7dbl = self.branch7x7dbl_5(branch7x7dbl) + + branch_pool = F.avg_pool2d(x, kernel_size=3, stride=1, padding=1) + branch_pool = self.branch_pool(branch_pool) + + outputs = [branch1x1, branch7x7, branch7x7dbl, branch_pool] + return torch.cat(outputs, 1) + + +class InceptionD(nn.Module): + def __init__(self, in_channels): + super(InceptionD, self).__init__() + self.branch3x3_1 = BasicConv2d(in_channels, 192, kernel_size=1) + self.branch3x3_2 = BasicConv2d(192, 320, kernel_size=3, stride=2) + + self.branch7x7x3_1 = BasicConv2d(in_channels, 192, kernel_size=1) + self.branch7x7x3_2 = BasicConv2d(192, 192, kernel_size=(1, 7), padding=(0, 3)) + self.branch7x7x3_3 = BasicConv2d(192, 192, kernel_size=(7, 1), padding=(3, 0)) + self.branch7x7x3_4 = BasicConv2d(192, 192, kernel_size=3, stride=2) + + def forward(self, x): + branch3x3 = self.branch3x3_1(x) + branch3x3 = self.branch3x3_2(branch3x3) + + branch7x7x3 = self.branch7x7x3_1(x) + branch7x7x3 = self.branch7x7x3_2(branch7x7x3) + branch7x7x3 = self.branch7x7x3_3(branch7x7x3) + branch7x7x3 = self.branch7x7x3_4(branch7x7x3) + + branch_pool = F.max_pool2d(x, kernel_size=3, stride=2) + outputs = [branch3x3, branch7x7x3, branch_pool] + return torch.cat(outputs, 1) + + +class InceptionE(nn.Module): + def __init__(self, in_channels): + super(InceptionE, self).__init__() + self.branch1x1 = BasicConv2d(in_channels, 320, kernel_size=1) + + self.branch3x3_1 = BasicConv2d(in_channels, 384, kernel_size=1) + self.branch3x3_2a = BasicConv2d(384, 384, kernel_size=(1, 3), padding=(0, 1)) + self.branch3x3_2b = BasicConv2d(384, 384, kernel_size=(3, 1), padding=(1, 0)) + + self.branch3x3dbl_1 = BasicConv2d(in_channels, 448, kernel_size=1) + self.branch3x3dbl_2 = BasicConv2d(448, 384, kernel_size=3, padding=1) + self.branch3x3dbl_3a = BasicConv2d(384, 384, kernel_size=(1, 3), padding=(0, 1)) + self.branch3x3dbl_3b = BasicConv2d(384, 384, kernel_size=(3, 1), padding=(1, 0)) + + self.branch_pool = BasicConv2d(in_channels, 192, kernel_size=1) + + def forward(self, x): + branch1x1 = self.branch1x1(x) + + branch3x3 = self.branch3x3_1(x) + branch3x3 = [ + self.branch3x3_2a(branch3x3), + self.branch3x3_2b(branch3x3), + ] + branch3x3 = torch.cat(branch3x3, 1) + + branch3x3dbl = self.branch3x3dbl_1(x) + branch3x3dbl = self.branch3x3dbl_2(branch3x3dbl) + branch3x3dbl = [ + self.branch3x3dbl_3a(branch3x3dbl), + self.branch3x3dbl_3b(branch3x3dbl), + ] + branch3x3dbl = torch.cat(branch3x3dbl, 1) + + branch_pool = F.avg_pool2d(x, kernel_size=3, stride=1, padding=1) + branch_pool = self.branch_pool(branch_pool) + + outputs = [branch1x1, branch3x3, branch3x3dbl, branch_pool] + return torch.cat(outputs, 1) + + +class InceptionAux(nn.Module): + def __init__(self, in_channels, num_classes): + super(InceptionAux, self).__init__() + self.conv0 = BasicConv2d(in_channels, 128, kernel_size=1) + self.conv1 = BasicConv2d(128, 768, kernel_size=5) + self.conv1.stddev = 0.01 + self.fc = nn.Linear(768, num_classes) + self.fc.stddev = 0.001 + + def forward(self, x): + # N x 768 x 17 x 17 + x = F.avg_pool2d(x, kernel_size=5, stride=3) + # N x 768 x 5 x 5 + x = self.conv0(x) + # N x 128 x 5 x 5 + x = self.conv1(x) + # N x 768 x 1 x 1 + # Adaptive average pooling + x = F.adaptive_avg_pool2d(x, (1, 1)) + # N x 768 x 1 x 1 + x = x.view(x.size(0), -1) + # N x 768 + x = self.fc(x) + # N x 1000 + return x + + +class BasicConv2d(nn.Module): + def __init__(self, in_channels, out_channels, **kwargs): + super(BasicConv2d, self).__init__() + self.conv = nn.Conv2d(in_channels, out_channels, bias=False, **kwargs) + self.bn = nn.BatchNorm2d(out_channels, eps=0.001) + + def forward(self, x): + x = self.conv(x) + x = self.bn(x) + return F.relu(x, inplace=True) diff --git a/models/resnet.py b/models/resnet.py new file mode 100644 index 0000000..183f1e6 --- /dev/null +++ b/models/resnet.py @@ -0,0 +1,303 @@ +import torch +import torch.nn as nn +import os + +__all__ = [ + "ResNet", + "resnet18", + "resnet34", + "resnet50", +] + + +def conv3x3(in_planes, out_planes, stride=1, groups=1, dilation=1): + """3x3 convolution with padding""" + return nn.Conv2d( + in_planes, + out_planes, + kernel_size=3, + stride=stride, + padding=dilation, + groups=groups, + bias=False, + dilation=dilation, + ) + + +def conv1x1(in_planes, out_planes, stride=1): + """1x1 convolution""" + return nn.Conv2d(in_planes, out_planes, kernel_size=1, stride=stride, bias=False) + + +class BasicBlock(nn.Module): + expansion = 1 + + def __init__( + self, + inplanes, + planes, + stride=1, + downsample=None, + groups=1, + base_width=64, + dilation=1, + norm_layer=None, + ): + super(BasicBlock, self).__init__() + if norm_layer is None: + norm_layer = nn.BatchNorm2d + if groups != 1 or base_width != 64: + raise ValueError("BasicBlock only supports groups=1 and base_width=64") + if dilation > 1: + raise NotImplementedError("Dilation > 1 not supported in BasicBlock") + # Both self.conv1 and self.downsample layers downsample the input when stride != 1 + self.conv1 = conv3x3(inplanes, planes, stride) + self.bn1 = norm_layer(planes) + self.relu = nn.ReLU(inplace=True) + self.conv2 = conv3x3(planes, planes) + self.bn2 = norm_layer(planes) + self.downsample = downsample + self.stride = stride + + def forward(self, x): + identity = x + + out = self.conv1(x) + out = self.bn1(out) + out = self.relu(out) + + out = self.conv2(out) + out = self.bn2(out) + + if self.downsample is not None: + identity = self.downsample(x) + + out += identity + out = self.relu(out) + + return out + + +class Bottleneck(nn.Module): + expansion = 4 + + def __init__( + self, + inplanes, + planes, + stride=1, + downsample=None, + groups=1, + base_width=64, + dilation=1, + norm_layer=None, + ): + super(Bottleneck, self).__init__() + if norm_layer is None: + norm_layer = nn.BatchNorm2d + width = int(planes * (base_width / 64.0)) * groups + # Both self.conv2 and self.downsample layers downsample the input when stride != 1 + self.conv1 = conv1x1(inplanes, width) + self.bn1 = norm_layer(width) + self.conv2 = conv3x3(width, width, stride, groups, dilation) + self.bn2 = norm_layer(width) + self.conv3 = conv1x1(width, planes * self.expansion) + self.bn3 = norm_layer(planes * self.expansion) + self.relu = nn.ReLU(inplace=True) + self.downsample = downsample + self.stride = stride + + def forward(self, x): + identity = x + + out = self.conv1(x) + out = self.bn1(out) + out = self.relu(out) + + out = self.conv2(out) + out = self.bn2(out) + out = self.relu(out) + + out = self.conv3(out) + out = self.bn3(out) + + if self.downsample is not None: + identity = self.downsample(x) + + out += identity + out = self.relu(out) + + return out + + +class ResNet(nn.Module): + def __init__( + self, + block, + layers, + num_classes=10, + zero_init_residual=False, + groups=1, + width_per_group=64, + replace_stride_with_dilation=None, + norm_layer=None, + ): + super(ResNet, self).__init__() + if norm_layer is None: + norm_layer = nn.BatchNorm2d + self._norm_layer = norm_layer + + self.inplanes = 64 + self.dilation = 1 + if replace_stride_with_dilation is None: + # each element in the tuple indicates if we should replace + # the 2x2 stride with a dilated convolution instead + replace_stride_with_dilation = [False, False, False] + if len(replace_stride_with_dilation) != 3: + raise ValueError( + "replace_stride_with_dilation should be None " + "or a 3-element tuple, got {}".format(replace_stride_with_dilation) + ) + self.groups = groups + self.base_width = width_per_group + + # CIFAR10: kernel_size 7 -> 3, stride 2 -> 1, padding 3->1 + self.conv1 = nn.Conv2d( + 3, self.inplanes, kernel_size=3, stride=1, padding=1, bias=False + ) + # END + + self.bn1 = norm_layer(self.inplanes) + self.relu = nn.ReLU(inplace=True) + self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1) + self.layer1 = self._make_layer(block, 64, layers[0]) + self.layer2 = self._make_layer( + block, 128, layers[1], stride=2, dilate=replace_stride_with_dilation[0] + ) + self.layer3 = self._make_layer( + block, 256, layers[2], stride=2, dilate=replace_stride_with_dilation[1] + ) + self.layer4 = self._make_layer( + block, 512, layers[3], stride=2, dilate=replace_stride_with_dilation[2] + ) + self.avgpool = nn.AdaptiveAvgPool2d((1, 1)) + self.fc = nn.Linear(512 * block.expansion, num_classes) + + for m in self.modules(): + if isinstance(m, nn.Conv2d): + nn.init.kaiming_normal_(m.weight, mode="fan_out", nonlinearity="relu") + elif isinstance(m, (nn.BatchNorm2d, nn.GroupNorm)): + nn.init.constant_(m.weight, 1) + nn.init.constant_(m.bias, 0) + + # Zero-initialize the last BN in each residual branch, + # so that the residual branch starts with zeros, and each residual block behaves like an identity. + # This improves the model by 0.2~0.3% according to https://arxiv.org/abs/1706.02677 + if zero_init_residual: + for m in self.modules(): + if isinstance(m, Bottleneck): + nn.init.constant_(m.bn3.weight, 0) + elif isinstance(m, BasicBlock): + nn.init.constant_(m.bn2.weight, 0) + + def _make_layer(self, block, planes, blocks, stride=1, dilate=False): + norm_layer = self._norm_layer + downsample = None + previous_dilation = self.dilation + if dilate: + self.dilation *= stride + stride = 1 + if stride != 1 or self.inplanes != planes * block.expansion: + downsample = nn.Sequential( + conv1x1(self.inplanes, planes * block.expansion, stride), + norm_layer(planes * block.expansion), + ) + + layers = [] + layers.append( + block( + self.inplanes, + planes, + stride, + downsample, + self.groups, + self.base_width, + previous_dilation, + norm_layer, + ) + ) + self.inplanes = planes * block.expansion + for _ in range(1, blocks): + layers.append( + block( + self.inplanes, + planes, + groups=self.groups, + base_width=self.base_width, + dilation=self.dilation, + norm_layer=norm_layer, + ) + ) + + return nn.Sequential(*layers) + + def forward(self, x): + x = self.conv1(x) + x = self.bn1(x) + x = self.relu(x) + x = self.maxpool(x) + + x = self.layer1(x) + x = self.layer2(x) + x = self.layer3(x) + x = self.layer4(x) + + x = self.avgpool(x) + x = x.reshape(x.size(0), -1) + x = self.fc(x) + + return x + + +def _resnet(arch, block, layers, pretrained, progress, device, **kwargs): + model = ResNet(block, layers, **kwargs) + if pretrained: + script_dir = os.path.dirname(__file__) + state_dict = torch.load( + script_dir + "/state_dicts/" + arch + ".pt", map_location=device + ) + model.load_state_dict(state_dict) + return model + + +def resnet18(pretrained=False, progress=True, device="cpu", **kwargs): + """Constructs a ResNet-18 model. + Args: + pretrained (bool): If True, returns a model pre-trained on ImageNet + progress (bool): If True, displays a progress bar of the download to stderr + """ + return _resnet( + "resnet18", BasicBlock, [2, 2, 2, 2], pretrained, progress, device, **kwargs + ) + + +def resnet34(pretrained=False, progress=True, device="cpu", **kwargs): + """Constructs a ResNet-34 model. + Args: + pretrained (bool): If True, returns a model pre-trained on ImageNet + progress (bool): If True, displays a progress bar of the download to stderr + """ + return _resnet( + "resnet34", BasicBlock, [3, 4, 6, 3], pretrained, progress, device, **kwargs + ) + + +def resnet50(pretrained=False, progress=True, device="cpu", **kwargs): + """Constructs a ResNet-50 model. + Args: + pretrained (bool): If True, returns a model pre-trained on ImageNet + progress (bool): If True, displays a progress bar of the download to stderr + """ + return _resnet( + "resnet50", Bottleneck, [3, 4, 6, 3], pretrained, progress, device, **kwargs + ) diff --git a/models/vgg.py b/models/vgg.py new file mode 100644 index 0000000..7dd9949 --- /dev/null +++ b/models/vgg.py @@ -0,0 +1,172 @@ +import os + +import torch +import torch.nn as nn + +__all__ = [ + "VGG", + "vgg11_bn", + "vgg13_bn", + "vgg16_bn", + "vgg19_bn", +] + + +class VGG(nn.Module): + def __init__(self, features, num_classes=10, init_weights=True): + super(VGG, self).__init__() + self.features = features + # CIFAR 10 (7, 7) to (1, 1) + # self.avgpool = nn.AdaptiveAvgPool2d((7, 7)) + self.avgpool = nn.AdaptiveAvgPool2d((1, 1)) + + self.classifier = nn.Sequential( + nn.Linear(512 * 1 * 1, 4096), + # nn.Linear(512 * 7 * 7, 4096), + nn.ReLU(True), + nn.Dropout(), + nn.Linear(4096, 4096), + nn.ReLU(True), + nn.Dropout(), + nn.Linear(4096, num_classes), + ) + if init_weights: + self._initialize_weights() + + def forward(self, x): + x = self.features(x) + x = self.avgpool(x) + x = x.view(x.size(0), -1) + x = self.classifier(x) + return x + + def _initialize_weights(self): + for m in self.modules(): + if isinstance(m, nn.Conv2d): + nn.init.kaiming_normal_(m.weight, mode="fan_out", nonlinearity="relu") + if m.bias is not None: + nn.init.constant_(m.bias, 0) + elif isinstance(m, nn.BatchNorm2d): + nn.init.constant_(m.weight, 1) + nn.init.constant_(m.bias, 0) + elif isinstance(m, nn.Linear): + nn.init.normal_(m.weight, 0, 0.01) + nn.init.constant_(m.bias, 0) + + +def make_layers(cfg, batch_norm=False): + layers = [] + in_channels = 3 + for v in cfg: + if v == "M": + layers += [nn.MaxPool2d(kernel_size=2, stride=2)] + else: + conv2d = nn.Conv2d(in_channels, v, kernel_size=3, padding=1) + if batch_norm: + layers += [conv2d, nn.BatchNorm2d(v), nn.ReLU(inplace=True)] + else: + layers += [conv2d, nn.ReLU(inplace=True)] + in_channels = v + return nn.Sequential(*layers) + + +cfgs = { + "A": [64, "M", 128, "M", 256, 256, "M", 512, 512, "M", 512, 512, "M"], + "B": [64, 64, "M", 128, 128, "M", 256, 256, "M", 512, 512, "M", 512, 512, "M"], + "D": [ + 64, + 64, + "M", + 128, + 128, + "M", + 256, + 256, + 256, + "M", + 512, + 512, + 512, + "M", + 512, + 512, + 512, + "M", + ], + "E": [ + 64, + 64, + "M", + 128, + 128, + "M", + 256, + 256, + 256, + 256, + "M", + 512, + 512, + 512, + 512, + "M", + 512, + 512, + 512, + 512, + "M", + ], +} + + +def _vgg(arch, cfg, batch_norm, pretrained, progress, device, **kwargs): + if pretrained: + kwargs["init_weights"] = False + model = VGG(make_layers(cfgs[cfg], batch_norm=batch_norm), **kwargs) + if pretrained: + script_dir = os.path.dirname(__file__) + state_dict = torch.load( + script_dir + "/state_dicts/" + arch + ".pt", map_location=device + ) + model.load_state_dict(state_dict) + return model + + +def vgg11_bn(pretrained=False, progress=True, device="cpu", **kwargs): + """VGG 11-layer model (configuration "A") with batch normalization + + Args: + pretrained (bool): If True, returns a model pre-trained on ImageNet + progress (bool): If True, displays a progress bar of the download to stderr + """ + return _vgg("vgg11_bn", "A", True, pretrained, progress, device, **kwargs) + + +def vgg13_bn(pretrained=False, progress=True, device="cpu", **kwargs): + """VGG 13-layer model (configuration "B") with batch normalization + + Args: + pretrained (bool): If True, returns a model pre-trained on ImageNet + progress (bool): If True, displays a progress bar of the download to stderr + """ + return _vgg("vgg13_bn", "B", True, pretrained, progress, device, **kwargs) + + +def vgg16_bn(pretrained=False, progress=True, device="cpu", **kwargs): + """VGG 16-layer model (configuration "D") with batch normalization + + Args: + pretrained (bool): If True, returns a model pre-trained on ImageNet + progress (bool): If True, displays a progress bar of the download to stderr + """ + return _vgg("vgg16_bn", "D", True, pretrained, progress, device, **kwargs) + + +def vgg19_bn(pretrained=False, progress=True, device="cpu", **kwargs): + """VGG 19-layer model (configuration 'E') with batch normalization + + Args: + pretrained (bool): If True, returns a model pre-trained on ImageNet + progress (bool): If True, displays a progress bar of the download to stderr + """ + return _vgg("vgg19_bn", "E", True, pretrained, progress, device, **kwargs) diff --git a/scripts/end-to-end-sh-scripts.txt b/scripts/end-to-end-sh-scripts.txt new file mode 100644 index 0000000..e69de29 diff --git a/scripts/end_to_end.sh b/scripts/end_to_end.sh index 30e9bb9..8006764 100755 --- a/scripts/end_to_end.sh +++ b/scripts/end_to_end.sh @@ -14,7 +14,7 @@ OUTPUT_PATH="../output/" SRC_PATH="../src/" LOG_PATH="./log/" SCRIPT1="../src/preprocess.py" -SCRIPT2="../src/profile.py" +SCRIPT2="../src/profiling.py" SCRIPT3="../src/split_data.py" SCRIPT4="../src/injections.py" SCRIPT5="../src/postprocess.py" diff --git a/src/dataloader.py b/src/dataloader.py new file mode 100755 index 0000000..f57d09d --- /dev/null +++ b/src/dataloader.py @@ -0,0 +1,94 @@ +import os +import torch +import torchvision +from PIL import Image +import torch.utils.data +from pycocotools.coco import COCO +from torchvision.models.detection.faster_rcnn import FastRCNNPredictor + +class COCODataset(torch.utils.data.Dataset): + def __init__(self, root, annotation, transforms=None): + self.root = root + self.transforms = transforms + self.coco = COCO(annotation) + self.ids = list(sorted(self.coco.imgs.keys())) + + def __getitem__(self, index): + # Own coco file + coco = self.coco + # Image ID + img_id = self.ids[index] + # List: get annotation id from coco + ann_ids = coco.getAnnIds(imgIds=img_id) + # Dictionary: target coco_annotation file for an image + coco_annotation = coco.loadAnns(ann_ids) + # path for input image + path = coco.loadImgs(img_id)[0]["file_name"] + # open the input image + img = Image.open(os.path.join(self.root, path)) + # number of objects in the image + num_objs = len(coco_annotation) + # Size of bbox (Rectangular) + areas = 0 + + labels = [] + for i in range(num_objs): + labels.append(coco_annotation[i]['category_id']) + labels = torch.as_tensor(labels, dtype=torch.int64) + + # Bounding boxes for objects + # In coco format, bbox = [xmin, ymin, width, height] + # In pytorch, the input should be [xmin, ymin, xmax, ymax] + boxes = [] + for i in range(num_objs): + xmin = coco_annotation[i]["bbox"][0] + ymin = coco_annotation[i]["bbox"][1] + xmax = xmin + coco_annotation[i]["bbox"][2] + ymax = ymin + coco_annotation[i]["bbox"][3] + + if xmin == xmax or ymin == ymax: + continue + else: + boxes.append([xmin, ymin, xmax, ymax]) + areas += (xmax-xmin) * (ymax-ymin) + boxes = torch.as_tensor(boxes, dtype=torch.float32) + + # Handle empty bounding boxes + if num_objs == 0: + boxes = torch.zeros((0, 4), dtype=torch.float32) + else: + boxes = torch.as_tensor(boxes, dtype=torch.float32) + # Labels (class: target) + + # Tensorise img_id + img_id = torch.tensor([img_id]) + + areas = torch.as_tensor(areas, dtype=torch.float32) + # Iscrowd + iscrowd = torch.zeros((num_objs,), dtype=torch.int64) + + # Annotation is in dictionary format + annotation = {} + annotation["boxes"] = boxes + annotation["labels"] = labels + annotation["image_id"] = img_id + annotation["area"] = areas + annotation["iscrowd"] = iscrowd + + if self.transforms is not None: + img = self.transforms(img) + + return img, annotation, index + + def __len__(self): + return len(self.ids) + +# In my case, just added ToTensor +def get_transform(): + custom_transforms = [] + custom_transforms.append(torchvision.transforms.ToTensor()) + return torchvision.transforms.Compose(custom_transforms) + +# collate_fn needs for batch +def collate_fn(batch): + return tuple(zip(*batch)) diff --git a/src/error_models_num_sys.py b/src/error_models_num_sys.py old mode 100644 new mode 100755 diff --git a/src/goldeneye.py b/src/goldeneye.py old mode 100644 new mode 100755 index 047127a..9a70c34 --- a/src/goldeneye.py +++ b/src/goldeneye.py @@ -50,7 +50,11 @@ def __init__( self.bits = kwargs.get("bits", 8) # the order of injecting within the goldeneye transformation - # 0 -> no injection, 1 -> between quantization and de-quantization, 2 -> after converting to the number system, 3 -> after dequantization, 4 -> after converting num sys + # 0 -> no injection, + # 1 -> between quantization and de-quantization, + # 2 -> after converting to the number system, + # 3 -> after dequantization, + # 4 -> after converting num sys def set_layer_max(self, data): self.LayerRanges = data diff --git a/src/injections.py b/src/injections.py old mode 100644 new mode 100755 index 3893554..c2c9ceb --- a/src/injections.py +++ b/src/injections.py @@ -1,15 +1,15 @@ import time -import torch.nn as nn - -# import pytorchfi.pytorchfi.error_models from util import * +from PIL import Image, ImageDraw +import torch.nn as nn from tqdm import tqdm from goldeneye import goldeneye +from torchvision.transforms.functional import to_pil_image +from profiling import calculate_precision_recall_f1,check_labels_tensors # from num_sys_class import * # sys.path.append("./pytorchfi") - def rand_neurons_batch(pfi_model, layer, shape, maxval, batchsize, function=-1): dim = len(shape) batch, layerArr, dim1, dim2, dim3, value = ([] for i in range(6)) @@ -43,13 +43,11 @@ def rand_neurons_batch(pfi_model, layer, shape, maxval, batchsize, function=-1): function=function, ) - - - if __name__ == "__main__": # Read in cmd line args check_args(sys.argv[1:]) + printArgs() if getDebug(): printArgs() @@ -79,6 +77,13 @@ def rand_neurons_batch(pfi_model, layer, shape, maxval, batchsize, function=-1): profile_path = getOutputDir() + "/networkProfiles/" + name + "/" data_subset_path = getOutputDir() + "/data_subset/" + name + "/" out_path = getOutputDir() + "/injections/" + name + "/" + out_saved_imgs_path = getOutputDir() + "/injections/" + name + "/inj_output_imgs" + + if os.path.exists(out_saved_imgs_path): + pass + else: + os.mkdir(out_saved_imgs_path) + image_set = "" if getTraining_en(): @@ -106,12 +111,16 @@ def rand_neurons_batch(pfi_model, layer, shape, maxval, batchsize, function=-1): workers=getWorkers(), ) model = getNetwork(getDNN(), getDataset()) - criterion = nn.CrossEntropyLoss(reduction="none") + criterion = torchvision.models.detection.fasterrcnn_resnet50_fpn().cuda() + criterion.train() + img_ids = 0 + #criterion = nn.CrossEntropyLoss(reduction="none") if getCUDA_en(): model = model.cuda() if getPrecision() == "FP16": model = model.half() + print(" Dataset & Model loaded....") model.eval() torch.no_grad() @@ -120,12 +129,18 @@ def rand_neurons_batch(pfi_model, layer, shape, maxval, batchsize, function=-1): if "IMAGENET" in getDataset(): baseH = 224 baseW = 224 + elif "CIFAR" in getDataset(): baseH = 32 baseW = 32 + elif "COCO" in getDataset(): + baseH = 480 + baseW = 640 + exp_bits = getBitwidth() - getRadix() - 1 # also INT for fixed point mantissa_bits = getRadix() # also FRAC for fixed point + goldeneye_model = goldeneye( model, getBatchsize(), @@ -170,20 +185,18 @@ def rand_neurons_batch(pfi_model, layer, shape, maxval, batchsize, function=-1): pbar.update(samples) # prep images - images, labels, img_ids, index = dataiter.next() - if getCUDA_en(): - labels = labels.cuda() - images = images.cuda() - if getPrecision() == "FP16": - images = images.half() - + for images, labels, index in dataiter: + images = list(img.cuda() for img in images) + labels = [{k: v.cuda() for k, v in t.items()} for t in labels] + + if getPrecision() == "FP16": + images = images.half() # inj_model = random_neuron_single_bit_inj_batched(pfi_model, ranges) # inj_model_locations = random_neuron_inj_batched(pfi_model, # min_val= abs(ranges[currLayer]) * -1, # max_val=abs(ranges[currLayer]), # ) - # injection locations with torch.no_grad(): inf_model = rand_neurons_batch(goldeneye_model, @@ -193,8 +206,6 @@ def rand_neurons_batch(pfi_model, layer, shape, maxval, batchsize, function=-1): getBatchsize(), function=goldeneye_model.apply_goldeneye_transformation ) - - # injection model # inf_model = goldeneye.declare_neuron_fi(batch=batch_inj, # layer=layer_inj, @@ -206,18 +217,47 @@ def rand_neurons_batch(pfi_model, layer, shape, maxval, batchsize, function=-1): # perform inference output_inj = inf_model(images) - output_argmax = torch.argmax(output_inj, dim=1) - output_inj_loss = criterion(output_inj, labels) - - # save results - layerInjects.append( - ( - index.tolist(), - output_argmax.data.tolist(), - output_inj_loss.tolist(), + out_loss = criterion(images,labels) + cls_loss = out_loss['loss_classifier'].item() + bbox_loss = out_loss['loss_box_reg'].item() + obj_loss = out_loss['loss_objectness'].item() + rpn_loss = out_loss['loss_rpn_box_reg'].item() + out_inj_lossess = [cls_loss, bbox_loss, obj_loss, rpn_loss] + + for idx, img in enumerate(images): + try: + img = to_pil_image(img) + draw = ImageDraw.Draw(img) + + inj_pred_bboxes = output_inj[idx]['boxes'].int().cpu() + inj_pred_labels = output_inj[idx]['labels'].cpu() + inj_pred_box_confs = output_inj[idx]['scores'] + gr_label = labels[idx]['labels'].cpu() + inj_correct_boxes = check_labels_tensors(gr_label.cpu(), inj_pred_labels.cpu())[0] + inj_wrong_boxes = check_labels_tensors(gr_label.cpu(), inj_pred_labels.cpu())[1] + + inj_precisions= calculate_precision_recall_f1(output_inj[idx]['boxes'],labels[idx]['boxes'], 0.60) + + for box, inj_pd_label, inj_score, in zip(inj_pred_bboxes, inj_pred_labels, inj_pred_box_confs): + if inj_score > 0.4: + draw.rectangle([(box[0], box[1]), (box[2], box[3])], outline="blue", width=4) + draw.text((box[0], box[1]), f"{inj_pd_label}: {inj_score:.2f}", fill="red") + img.save(f"{out_saved_imgs_path}/output_{index[idx]}_{str(currLayer)}{inj_score:.2f}.png") + except: + continue + + layerInjects.append( + ( + index, + inj_pred_bboxes, + inj_pred_labels, + inj_pred_box_confs, + inj_precisions, + inj_correct_boxes, + inj_wrong_boxes, + out_inj_lossess, + ) ) - ) - samples += getBatchsize() torch.cuda.empty_cache() # print("") diff --git a/src/num_sys.cpp b/src/num_sys.cpp old mode 100644 new mode 100755 diff --git a/src/num_sys.h b/src/num_sys.h old mode 100644 new mode 100755 diff --git a/src/num_sys_class.py b/src/num_sys_class.py old mode 100644 new mode 100755 diff --git a/src/num_sys_helper.cpp b/src/num_sys_helper.cpp old mode 100644 new mode 100755 diff --git a/src/num_sys_helper.h b/src/num_sys_helper.h old mode 100644 new mode 100755 diff --git a/src/perf_measurement.py b/src/perf_measurement.py old mode 100644 new mode 100755 diff --git a/src/postprocess.py b/src/postprocess.py old mode 100644 new mode 100755 index e9ee767..8296d60 --- a/src/postprocess.py +++ b/src/postprocess.py @@ -1,11 +1,13 @@ from util import * -import concurrent.futures import math import pandas as pd from tqdm import tqdm +import concurrent.futures + +MAX_INJ = 10 -MAX_INJ = 0 def setMax(value): + global MAX_INJ MAX_INJ = value @@ -13,7 +15,23 @@ def getMax(): global MAX_INJ return MAX_INJ +def subtract_losses(list1, list2): + if len(list1) != len(list2): + raise ValueError("Lists must have the same length") + absolute_diff_loss = [abs(val1 - val2) for val1, val2 in zip(list1, list2)] + return absolute_diff_loss + +def calculate_avg_detal_loss(list1, list2): + if len(list1) != len(list2): + raise ValueError("Lists must have the same length") + + abs_diffs = [abs(val1 - val2) for val1, val2 in zip(list1, list2)] + avg_abs_diffs = sum(abs_diffs) / len(abs_diffs) + + return avg_abs_diffs + def _layer_file_processing(data_in): + inj_path, curr_layer, golden_data = data_in layerFileName = "layer" + str(curr_layer) layer_inj_data = load_file(inj_path + layerFileName) @@ -23,13 +41,14 @@ def _layer_file_processing(data_in): _layer_loss = 0 SIZE_ = getMax() nans = 0 - LayerData = [] PRINT_INJ_INFO = True # NOT Optimal: should compute running average and running variance instead.. # average + for batch in range(len(layer_inj_data)): + if total_injections >= SIZE_: break @@ -37,42 +56,58 @@ def _layer_file_processing(data_in): # batch_img_id, batch_inj_bit, batch_argmax, batch_inj_loss = fmap_inj_data[batch] # else: # batch_img_id, batch_inj_H, batch_inj_W, batch_inj_val, batch_argmax, batch_inj_loss = fmap_inj_data[batch] - batch_img_id, batch_argmax, batch_inj_loss = layer_inj_data[batch] + + #batch_img_id, batch_argmax, batch_inj_loss = layer_inj_data[batch] + batch_img_id, batch_inj_pred_bboxes, batch_inj_pred_labels, batch_box_confs, batch_inj_precisions, batch_inj_correct_boxes, batch_inj_wrong_boxes, batch_inj_losses = layer_inj_data[batch] for injection in range(len(batch_img_id)): if total_injections >= SIZE_: break total_injections += 1 + # injection info img_id = batch_img_id[injection] - inj_loss = batch_inj_loss[injection] - inj_label = batch_argmax[injection] + inj_pred_box_cords = [batch_inj_pred_bboxes] + inj_pred_box_confs = [batch_box_confs] + inj_precisions = [batch_inj_precisions] + inj_pred_labels = [batch_inj_pred_labels] + inj_correct_boxes = batch_inj_correct_boxes + inj_wrong_boxes = batch_inj_wrong_boxes + inj_losses = [batch_inj_losses] - if (math.isnan(inj_loss)): - nans += 1 - continue + # if (math.isnan(inj_losses)): + # nans += 1 + # continue + """img_id (int), gr_labels [] , pred_labels [] , gr_boxes_count (int), pred_boxes_count (int), correct_box_count (int), wrong_box_count (int), + box_conf (float), cls_loss (float), bbox_loss (float), obj_loss (float), rpn_loss (float).""" # golden info - gold_inf, gold_label, gold_conf, gold_top2diff, gold_loss = golden_data[img_id] + #gold_inf, gold_label, gold_conf, gold_top2diff, gold_loss = golden_data[img_id] + + true_box_coords, gold_pred_box_coords, gr_label, gold_pred_labels, bbox_ious, gold_pred_box_confs, gold_precisions, recalls, f1_scores , gr_boxes, pred_boxes,gold_correct_boxes, gold_wrong_boxes, cls_loss, bbox_loss, obj_loss, rpn_loss = golden_data[img_id] + gold_losses = [cls_loss, bbox_loss, obj_loss, rpn_loss] # compare golden info with injection info - if(inj_label != gold_label): + if(inj_correct_boxes != gold_correct_boxes): _layer_mismatches += 1 # compare loss - _layer_loss += abs(gold_loss - inj_loss) + _layer_loss = subtract_losses(gold_losses, inj_losses) + print("Absolute_layer_loss: ", _layer_loss) - injData = (img_id, gold_conf, gold_top2diff, gold_label, inj_label, gold_loss, inj_loss) + injData = (img_id, gold_pred_box_coords, inj_pred_box_cords, gold_pred_box_confs, inj_pred_box_confs, gold_precisions, inj_precisions, gold_pred_labels, inj_pred_labels, gold_correct_boxes, inj_correct_boxes, gold_wrong_boxes, inj_wrong_boxes, gold_losses, inj_losses) LayerData.append(injData) - ave_delta_loss = _layer_loss / (total_injections - nans ) - + # avg_delta_loss = _layer_loss/total_injections + # print("avg_delta_loss: ", avg_delta_loss) + # # standard dev total_injections = 0 var_sum = 0.0 nans = 0 + for batch in range(len(layer_inj_data)): if total_injections >= SIZE_: break @@ -81,27 +116,41 @@ def _layer_file_processing(data_in): # batch_img_id, batch_inj_bit, batch_argmax, batch_inj_loss = fmap_inj_data[batch] # else: # batch_img_id, batch_inj_H, batch_inj_W, batch_inj_val, batch_argmax, batch_inj_loss = fmap_inj_data[batch] - batch_img_id, batch_argmax, batch_inj_loss = layer_inj_data[batch] - + batch_img_id, batch_inj_pred_bboxes, batch_inj_pred_labels, batch_box_confs, batch_inj_precisions, batch_inj_correct_boxes, batch_inj_wrong_boxes, batch_inj_losses = layer_inj_data[batch] + for injection in range(len(batch_img_id)): if total_injections >= SIZE_: break total_injections += 1 + # injection info img_id = batch_img_id[injection] - inj_loss = batch_inj_loss[injection] - inj_label = batch_argmax[injection] - - if (math.isnan(inj_loss)): - nans += 1 - continue + inj_pred_box_cords = [batch_inj_pred_bboxes] + inj_pred_box_confs = [batch_box_confs] + inj_precisions = [batch_inj_precisions] + inj_pred_labels = [batch_inj_pred_labels] + inj_correct_boxes = batch_inj_correct_boxes + inj_wrong_boxes = batch_inj_wrong_boxes + inj_losses = [batch_inj_losses] + + # if (math.isnan(inj_losses)): + # nans += 1 + # continue # golden info - gold_inf, gold_label, gold_conf, gold_top2diff, gold_loss = golden_data[img_id] + """img_id (int), gr_labels [] , pred_labels [] , gr_boxes_count (int), pred_boxes_count (int), correct_box_count (int), wrong_box_count (int), + box_conf (float), cls_loss (float), bbox_loss (float), obj_loss (float), rpn_loss (float).""" + + true_box_coords, gold_pred_box_coords, gr_label, gold_pred_labels, bbox_ious, gold_pred_box_confs, gold_precisions, recalls, f1_scores , gr_boxes, pred_boxes,gold_correct_boxes, gold_wrong_boxes, cls_loss, bbox_loss, obj_loss, rpn_loss = golden_data[img_id] + gold_losses = [cls_loss, bbox_loss, obj_loss, rpn_loss] # compare loss to average_loss - layer_for_var = abs(gold_loss - inj_loss) - ave_delta_loss + #calculate_avg_detal_loss + + #layer_for_var = abs(gold_losses[injection] - inj_losses[injection]) - avg_delta_loss + layer_for_var = subtract_losses(gold_losses, inj_losses) + #layer_for_var = subtract_losses(layer_for_var, avg_delta_loss) var_sum += (layer_for_var * layer_for_var) if nans > 0: @@ -115,22 +164,33 @@ def _layer_file_processing(data_in): fileName = inj_path + "layer" + str(curr_layer) + "_detailed" f_det = open(fileName + ".csv", "w+") - outputString = "Injection, ImgID, GoldConf, GoldTop2Diff, GoldLabel, InjLabel, GoldLoss, InjLoss, DeltaLoss\n" + outputString = "img_id, gold_pred_box_coords, inj_pred_box_cords, gold_pred_box_confs, inj_pred_box_confs, gold_precisions, inj_precisions, gold_pred_labels, inj_pred_labels, gold_correct_boxes, inj_correct_boxes, gold_wrong_boxes, inj_wrong_boxes, gold_losses, inj_losses\n" f_det.write(outputString) count = 0 for values in LayerData: - img_id, gold_conf, gold_top2diff, gold_label, inj_label, gold_loss, inj_loss = values - outputString = "%d, %d, %f, %f, %d, %d, %f, %f, %f\n" %(count, - img_id, - gold_conf, - gold_top2diff, - gold_label, - inj_label, - gold_loss, - inj_loss, - abs(gold_loss - inj_loss), - ) + img_id, gold_pred_box_coords, inj_pred_box_cords, gold_pred_box_confs, inj_pred_box_confs, gold_precisions, inj_precisions, gold_pred_labels, inj_pred_labels, gold_correct_boxes, inj_correct_boxes, gold_wrong_boxes, inj_wrong_boxes, gold_losses, inj_losses = values + + outputString = "%d, %d, %s, %s, %.2f, %.2f, %.2f, %s, %s, %s, %s, %s, %s, %s, %s, %.2f, %.2f\n" % ( + count, + img_id, + str(gold_pred_box_coords), + str(inj_pred_box_cords), + gold_pred_box_confs, + inj_pred_box_confs, + gold_precisions, + str(inj_precisions), + str(gold_pred_labels), + str(inj_pred_labels), + str(gold_correct_boxes), + str(inj_correct_boxes), + str(gold_wrong_boxes), + str(inj_wrong_boxes), + str(gold_losses), + str(inj_losses), + subtract_losses(gold_losses, inj_losses), + ) + f_det.write(outputString) count += 1 f_det.close() @@ -140,11 +200,12 @@ def _layer_file_processing(data_in): df.to_pickle(fileName + ".pkl") - return curr_layer, _layer_mismatches, ave_delta_loss, std_delta_loss + return curr_layer, _layer_mismatches, std_delta_loss if __name__ == "__main__": # Read in cmd line args check_args(sys.argv[1:]) + printArgs() if getDebug(): printArgs() # PATHS @@ -187,36 +248,39 @@ def _layer_file_processing(data_in): # Data Processing processing_tuple = [] + + torch.multiprocessing.set_start_method('spawn') with concurrent.futures.ProcessPoolExecutor(max_workers=getWorkers()) as executor: # prep parallel script data for layer in tqdm(range(LAYERS)): processing_tuple.append((inj_path, layer, golden_data)) - - + # launch in parallel results = executor.map(_layer_file_processing, processing_tuple) - - print("FINISHED") + + print("[INFO] Completed Postprocessing..") # split up for write to disk layer_mismatches = [] layer_loss = [] layer_loss_std = [] layer_counter = 0 + for i in results: layer_i, mismatches_i, loss_i, std_loss_i = i assert(layer_i == layer_counter) layer_mismatches.append([layer_i, mismatches_i]) layer_loss.append([layer_i, loss_i]) layer_loss_std.append([layer_i, std_loss_i]) - layer_counter += 1 + print("[INFO] Saving all data....") # save results save_data(out_path, "data_mismatches_" + str(INJ), layer_mismatches) save_data(out_path, "data_loss_" + str(INJ), layer_loss) f = open(out_path + "mismatches_" + str(INJ) + ".csv", "w+") + for i in range(len(layer_mismatches)): outputString = "%d, %d\n" %(i, layer_mismatches[i][1]) f.write(outputString) @@ -231,6 +295,7 @@ def _layer_file_processing(data_in): f = open(out_path + "loss_and_mismatches_" + str(INJ) + ".csv", "w+") outputString = "fmap_id, Mismatches, Loss, Sample_Size, STD_Loss, Error_Loss, Proportion, error_mismatches_proportion,error_mismatches\n" f.write(outputString) + for i in range(len(layer_loss)): z = 2.576 # Z value for 99% confidence error_loss_i = (z * layer_loss_std[i][1]) / math.sqrt(INJ) diff --git a/src/preprocess.py b/src/preprocess.py old mode 100644 new mode 100755 index 71a1d3e..ccb4033 --- a/src/preprocess.py +++ b/src/preprocess.py @@ -15,14 +15,15 @@ def gather_min_max_per_layer(model, data_iter, batch_size, precision="FP16", cud if cuda_en: layer_max = layer_max.cuda() layer_min = layer_min.cuda() + if precision == "FP16": layer_max = layer_max.half() layer_min = layer_min.half() - # register forward hook to the model handles = [] for param in model.modules(): + #or isinstance(param, nn.BatchNorm2d) or isinstance(param, nn.ReLU) or isinstance(param, nn.MaxPool2d) if isinstance(param, nn.Conv2d) or isinstance(param, nn.Linear): handles.append(param.register_forward_hook(save_activations)) @@ -34,9 +35,10 @@ def gather_min_max_per_layer(model, data_iter, batch_size, precision="FP16", cud # prepare the next batch for inference images, labels = input_data - if cuda_en: - images = images.cuda() - labels = labels.cuda() + # prepare the next batch for inference as an tuple for COCO Dataset + images = list(img for img in images) + labels = [{k: v for k, v in t.items()} for t in labels] + if precision == "FP16": images = images.half() @@ -50,11 +52,13 @@ def gather_min_max_per_layer(model, data_iter, batch_size, precision="FP16", cud .cuda() .half() ) + max_vals = ( torch.Tensor(list(map(lambda layer: layer.max().item(), activations))) .cuda() .half() ) + if batch_num == 0: layer_max = max_vals layer_min = min_vals @@ -87,6 +91,7 @@ def gather_min_max_per_layer(model, data_iter, batch_size, precision="FP16", cud # load data and model dataiter = load_dataset(getDataset(), getBatchsize(), workers = getWorkers(), training=True, include_id=False) + model = getNetwork(getDNN(), getDataset()) model.eval() torch.no_grad() @@ -111,4 +116,4 @@ def gather_min_max_per_layer(model, data_iter, batch_size, precision="FP16", cud for i in range(len(ranges)): outputString = "%d, %f\n" % (i, ranges[i]) f.write(outputString) - f.close() \ No newline at end of file + f.close() diff --git a/src/profiling.py b/src/profiling.py new file mode 100755 index 0000000..160f16c --- /dev/null +++ b/src/profiling.py @@ -0,0 +1,337 @@ +import os +import cv2 +from util import * +from PIL import Image, ImageDraw +import pandas as pd +import torch.nn as nn +from tqdm import tqdm +from goldeneye import * +from torchvision.transforms.functional import to_pil_image +from torchvision.models.detection import FasterRCNN_ResNet50_FPN_Weights, FasterRCNN_ResNet50_FPN_V2_Weights +from torchvision.models.detection.faster_rcnn import FastRCNNPredictor, fasterrcnn_resnet50_fpn_v2 + +def check_labels_tensors(tensor1, tensor2): + # Convert tensors to sets + set1 = set(tensor1.numpy()) + set2 = set(tensor2.numpy()) + + # Find the intersection of sets (matched elements) + intersection = set1.intersection(set2) + + # Count the number of matched elements (including duplicates) + ground_truth_boxes = sum(min(tensor1.numpy().tolist().count(element), + tensor2.numpy().tolist().count(element)) + for element in intersection) + + # Count the number of unmatched elements + wrong_boxes = abs(len(tensor2) - ground_truth_boxes) + return (ground_truth_boxes, wrong_boxes) + + +def make_prediction(model, img, threshold): + model.eval() + preds = model(img) + for id in range(len(preds)): + idx_list = [] + + for idx, score in enumerate(preds[id]['scores']): + if score > threshold: + idx_list.append(idx) + + preds[id]['boxes'] = preds[id]['boxes'][idx_list] + preds[id]['labels'] = preds[id]['labels'][idx_list] + preds[id]['scores'] = preds[id]['scores'][idx_list] + + return preds + +def calculate_iou(box1, box2): + # Calculate coordinates of intersection + x_left = max(box1[0], box2[0]) + y_top = max(box1[1], box2[1]) + x_right = min(box1[2], box2[2]) + y_bottom = min(box1[3], box2[3]) + + # Calculate area of intersection + intersection_area = max(0, x_right - x_left) * max(0, y_bottom - y_top) + + # Calculate areas of the boxes + box1_area = (box1[2] - box1[0]) * (box1[3] - box1[1]) + box2_area = (box2[2] - box2[0]) * (box2[3] - box2[1]) + + # Calculate the IoU + iou = intersection_area / float(box1_area + box2_area - intersection_area) + + return iou + +def calculate_iou_for_all_boxes(gt_boxes, pred_boxes, threshold=0.50): + iou_scores = [] + for box1 in gt_boxes: + iou_scores_row = [] + for box2 in pred_boxes: + iou_score = calculate_iou(box1, box2) + if iou_score >= threshold: + iou_scores_row.append(iou_score) + if iou_scores_row: + iou_scores.append(iou_scores_row) + return iou_scores + +def calculate_iou_p(box1, box2): + x1, y1, x2, y2 = box1 + x1_, y1_, x2_, y2_ = box2 + xA = torch.max(x1, x1_) + yA = torch.max(y1, y1_) + xB = torch.min(x2, x2_) + yB = torch.min(y2, y2_) + inter_area = torch.clamp(xB - xA, min=0) * torch.clamp(yB - yA, min=0) + box1_area = (x2 - x1) * (y2 - y1) + box2_area = (x2_ - x1_) * (y2_ - y1_) + iou = inter_area / (box1_area + box2_area - inter_area) + return iou + +def calculate_precision_recall_f1(predicted_boxes, true_boxes, iou_threshold): + precisions = [] + recalls = [] + f1_scores = [] + + for i in range(predicted_boxes.size(0)): + true_positives = 0 + false_positives = 0 + total_true_boxes = true_boxes.size(0) + total_predicted_boxes = predicted_boxes.size(0) + + if total_true_boxes == 0 or total_predicted_boxes == 0: + # Handle cases where there are no true boxes or predicted boxes + precision = 0.0 + recall = 0.0 + f1_score = 0.0 + else: + for j in range(total_predicted_boxes): + matched = False + for k in range(total_true_boxes): + iou = calculate_iou(predicted_boxes[j], true_boxes[k]) + if torch.max(iou) >= iou_threshold: + true_positives += 1 + matched = True + break + + if not matched: + false_positives += 1 + + precision = true_positives / (true_positives + false_positives) + recall = true_positives / total_true_boxes + f1_score = 2 * (precision * recall) / (precision + recall) + + precisions.append(precision) + recalls.append(recall) + f1_scores.append(f1_score) + + return precisions, recalls, f1_scores +@torch.no_grad() +def gather_golden(goldeneye, data_iter, cuda_en=True, precision='FP32', verbose=False, debug=False,out_saved_imgs_path=""): + golden_data = {} + good_boxes = 0 + bad_boxes = 0 + + criterion = torchvision.models.detection.fasterrcnn_resnet50_fpn( + weights=FasterRCNN_ResNet50_FPN_Weights.DEFAULT).cuda() + criterion.train() + + for images, labels, index in tqdm(data_iter): + inf_model = goldeneye.declare_neuron_fi(function=goldeneye.apply_goldeneye_transformation) + if precision == 'FP16': + images = images.half() + + images = list(img.cuda() for img in images) + labels = [{k: v.cuda() for k, v in t.items()} for t in labels] + + with torch.no_grad(): + # with torch.autocast(device_type='cuda', dtype=torch.float16# ): + out_pred = make_prediction(inf_model, images, 0.35) + out_loss = criterion(images, labels) + + cls_loss = out_loss['loss_classifier'].item() + bbox_loss = out_loss['loss_box_reg'].item() + obj_loss = out_loss['loss_objectness'].item() + rpn_loss = out_loss['loss_rpn_box_reg'].item() + + for idx, img in enumerate(images): + try: + img = to_pil_image(img) + draw = ImageDraw.Draw(img) + true_box_corrds = labels[idx]['boxes'].int().cpu() + pred_box_corrds = out_pred[idx]['boxes'].int().cpu() + pred_labels = out_pred[idx]['labels'].cpu() + gr_label = labels[idx]['labels'].cpu() + pred_box_conf = out_pred[idx]['scores'] + + precisions, recalls, f1_scores = calculate_precision_recall_f1(out_pred[idx]['boxes'], labels[idx]['boxes'],0.60) + + for box, pd_label, score, in zip(pred_box_corrds, pred_labels, pred_box_conf): + if score > 0.4: + draw.rectangle([(box[0], box[1]), (box[2], box[3])], outline="blue", width=4) + draw.text((box[0], box[1]), f"{pd_label}: {score:.2f}", fill="red") + img.save(f"{out_saved_imgs_path}/output_{index[idx]}.png") + + # for i in range(len(precisions)): + # print( + # f"Image {i + 1} - Precision: {precisions[i]:.2f}, Recall: {recalls[i]:.2f}, F1 Score: {f1_scores[i]:.2f}") + except: + continue + + correct_boxes = check_labels_tensors(gr_label.cpu(), pred_labels.cpu())[0] + good_boxes += correct_boxes + + wrong_boxes = check_labels_tensors(gr_label.cpu(), pred_labels.cpu())[1] + bad_boxes += wrong_boxes + + gr_boxes_list = labels[idx]['boxes'].int().cpu().tolist() + pred_boxes_list = out_pred[idx]['boxes'].int().cpu().tolist() + + gr_boxes = len(gr_label) + pred_boxes = len(pred_labels) + + bbox_iou = calculate_iou_for_all_boxes(gr_boxes_list, pred_boxes_list) + bbox_ious = [] + + for b_iou in bbox_iou: + for c_iou in b_iou: + if c_iou == 0.0: + continue + bbox_ious.append(c_iou) + """img_id (list), true_box_corrds(list), pred_box_corrds (list), gr_labels [ ] , pred_labels [ ] , bbox_ious, gr_boxes_count (int), pred_boxes_count (int), precisions, recall, correct_box_count (int), wrong_box_count (int), box_conf (float), cls_loss (float), bbox_loss (float), obj_loss (float), rpn_loss (float).""" + img_id = index[idx] + img_tuple = (true_box_corrds.tolist(),pred_box_corrds.tolist(), gr_label.tolist(), pred_labels.tolist(), bbox_iou, pred_box_conf.tolist(), precisions, recalls, f1_scores , gr_boxes, pred_boxes,correct_boxes, wrong_boxes, cls_loss, bbox_loss, obj_loss, rpn_loss) + assert (img_id not in golden_data) # we shouldn't have duplicates in the golden data + golden_data[img_id] = img_tuple + torch.cuda.empty_cache() + + total_boxes = good_boxes + bad_boxes + return golden_data, good_boxes, bad_boxes, total_boxes + +"""img_id (int), gr_labels [ ] , pred_labels [ ] , gr_boxes_count (int), pred_boxes_count (int),precisions, recalls, f1_scores ,correct_box_count (int), wrong_box_count (int), box_conf (float), cls_loss (float), bbox_loss (float), obj_loss (float), rpn_loss (float).""" + +def save_data_df(path, file_name, data): + if not os.path.exists(path): + os.makedirs(path) + + output = path + file_name + df = pd.DataFrame.from_dict(data, orient='index').reset_index() + df.columns = ['img_id', 'true_box_coords', 'pred_box_coords', 'gr_labels', 'pred_labels', 'bbox_ious', 'pred_box_confs', 'precisions', 'recalls', 'f1-scores','gr_boxes', + 'pred_boxes','correct_boxes', 'wrong_boxes', 'cls_loss', 'bbox_loss', 'obj_loss','rpn_loss'] + df.to_pickle(output + ".df") + df.to_csv(output + ".csv", index=False) + return df + +if __name__ == '__main__': + + # read in cmd line args + check_args(sys.argv[1:]) + if getDebug(): + printArgs() + + # common variables + range_name = getDNN() + "_" + getDataset() + range_path = getOutputDir() + "/networkRanges/" + range_name + "/" + + if getFormat() == "INT": + format = "INT" + quant_en = True + bitwidth_fp = 32 + else: + format = getFormat() + bitwidth_fp = getBitwidth() + quant_en = False + + name = getDNN() + "_" + getDataset() + "_real" + getPrecision() + "_sim" + format + "_bw" + str(bitwidth_fp) \ + + "_r" + str(getRadix()) + "_bias" + str(getBias()) + # if getQuantize_en(): name += "_" + "quant" + out_path = getOutputDir() + "/networkProfiles/" + name + "/" + subset_path = getOutputDir() + "/data_subset/" + name + "/" + imgs_out_folder = out_path+ "/output_images/" + + if os.path.exists(imgs_out_folder): + pass + else: + os.mkdir(imgs_out_folder) + # get ranges + ranges = load_file(range_path + "ranges_trainset_layer") + + # load data and model + dataiter = load_dataset(getDataset(), getBatchsize(), workers=getWorkers()) + model = getNetwork(getDNN(), getDataset()) + print(" Dataset & Model loaded....") + model.eval() + torch.no_grad() + + exp_bits = getBitwidth() - getRadix() - 1 # also INT for fixed point + mantissa_bits = getRadix() # getBitwidth() - exp_bits - 1 # also FRAC for fixed point + + # no injections during profiling + assert (getInjections() == -1) + assert (getInjectionsLocation() == 0) + + # init PyTorchFI + baseC = 3 + if "IMAGENET" in getDataset(): + baseH = 224 + baseW = 224 + elif "CIFAR" in getDataset(): + baseH = 32 + baseW = 32 + + elif "COCO" in getDataset(): + baseH = 480 + baseW = 640 + + goldeneye_model = goldeneye( + model, + getBatchsize(), + input_shape=[baseC, baseH, baseW], + layer_types=[nn.Conv2d, nn.Linear], + use_cuda=getCUDA_en(), + + # number format + signed=True, + num_sys=getNumSysName(getFormat(), + bits=bitwidth_fp, + radix_up=exp_bits, + radix_down=mantissa_bits, + bias=getBias()), + # quantization + quant=quant_en, + layer_max=ranges, + bits=getBitwidth(), + qsigned=True, + + inj_order=getInjectionsLocation(), + ) + + # Golden data gathering + golden_data, good_boxes, bad_boxes, total_boxes = gather_golden(goldeneye_model, dataiter, + getBatchsize(), precision=getPrecision(), + verbose=getVerbose(), debug=getDebug(), out_saved_imgs_path=imgs_out_folder) + # Golden data gathering + output_name = "golden_data" + save_data(out_path, output_name, golden_data) + df = save_data_df(out_path, output_name, golden_data) + + # Print Summary Statistics + summaryDetails = "" + summaryDetails += "===========================================\n" + summaryDetails += "%s\n" % (name) + summaryDetails += "Accuracy based on Pred_True Boxes: \t%0.2f%%\n" % (good_boxes / total_boxes * 100.0) + # summaryDetails += "Ave Conf: \t%0.2f%%\n" % (df["box_conf"].mean()) + summaryDetails += "===========================================\n" + + # save stats + stats_file = open(out_path + "stats.txt", "w+") + n = stats_file.write(summaryDetails) + stats_file.close() + + if getVerbose(): + print(summaryDetails) + print("===========================================") + print(name) + print("Accuracy based on Pred_True Boxes: \t%0.2f%%" % (good_boxes / total_boxes * 100.0)) + # print("Ave Conf: \t%0.2f%%" % (df["box_conf"].mean())) + print("===========================================") diff --git a/src/split_data.py b/src/split_data.py old mode 100644 new mode 100755 index fd828f2..72cded0 --- a/src/split_data.py +++ b/src/split_data.py @@ -2,7 +2,7 @@ ''' Randomizes and returns two lists. Split is between 0-1, and refers to the size of the rank set. -Example, .8 means 80/20 split +Example, 0.8 means 80/20 split ''' def gen_sets(golden_indices, split): total = len(golden_indices) @@ -37,13 +37,13 @@ def gen_sets(golden_indices, split): netProfilePath = getOutputDir() + "/networkProfiles/" + name + "/" outPath = getOutputDir() + "/data_subset/" + name + "/" golden_data = load_file(netProfilePath + "golden_data") + split_ratio = 0.8 - split_ratio = .8 - # generate an Analysis Set (AS) and Deployment Set (DS) if "IMAGENET" in getDataset(): images_base = list(range(0,50000)) elif "CIFAR" in getDataset(): images_base = list(range(0,10000)) - + elif "COCO" in getDataset(): images_base = list(range(0, 4999)) + random.seed(9001) analysis_set, deployment_set= gen_sets(images_base, split_ratio) save_data(outPath, "analysis_set", analysis_set) @@ -51,17 +51,22 @@ def gen_sets(golden_indices, split): random.seed() #back to randomness # generate a list from the correct images in AS and DS - # Also drop images where top2diff is 0 ASgoodImgs = [] DSgoodImgs = [] - for i in analysis_set: - if golden_data[i][0] == golden_data[i][1] and golden_data[i][3] > 0 : - ASgoodImgs.append(i) - for i in deployment_set: - if golden_data[i][0] == golden_data[i][1] and golden_data[i][3] > 0 : - DSgoodImgs.append(i) + try: + for i in analysis_set: + if golden_data[i][12] < 5: + ASgoodImgs.append(i) + except: + pass + try: + for i in deployment_set: + if golden_data[i][12] < 5: + DSgoodImgs.append(i) + except: + pass save_data(outPath, "rank_set_good", ASgoodImgs) save_data(outPath, "test_set_good", DSgoodImgs) diff --git a/src/sweep_num_formats.py b/src/sweep_num_formats.py old mode 100644 new mode 100755 diff --git a/src/test_goldeneye.py b/src/test_goldeneye.py old mode 100644 new mode 100755 diff --git a/src/test_neuron_num_sys.py b/src/test_neuron_num_sys.py old mode 100644 new mode 100755 diff --git a/src/testing_postprocess.py b/src/testing_postprocess.py new file mode 100644 index 0000000..7059407 --- /dev/null +++ b/src/testing_postprocess.py @@ -0,0 +1,5 @@ +from util import * +import math +import pandas as pd +from tqdm import tqdm +import concurrent.futures diff --git a/src/util.py b/src/util.py old mode 100644 new mode 100755 index c4682a7..e871b1b --- a/src/util.py +++ b/src/util.py @@ -1,21 +1,24 @@ -import sys, os, bz2, random -import argparse -import pickle as cPickle -import pandas as pd +import timm import torch +import dataloader +import argparse import torchvision +import numpy as np +import pickle as cPickle +from num_sys_class import * +import sys, os, bz2, random import torchvision.transforms as transforms import torchvision.datasets as datasets import torchvision.models as models -import timm -import numpy as np -from num_sys_class import * -from othermodels import resnet, vgg, cifar10_nn + +from torchvision.models.detection.faster_rcnn import FastRCNNPredictor, fasterrcnn_resnet50_fpn_v2 +from torchvision.models.detection import FasterRCNN_ResNet50_FPN_Weights, FasterRCNN_ResNet50_FPN_V2_Weights + ''' Environment Variables ''' -DATASETS = os.environ['ML_DATASETS'] +DATASETS = '' ''' Helper functions to parse input @@ -44,106 +47,25 @@ def check_args(args=None): parser = argparse.ArgumentParser(description='Arguments') - parser.add_argument('-b', '--batchsize', - help='Batch Size for inference', - type=int, - required='True', - default=8) - - parser.add_argument('-n', '--dnn', - help='Neural network', - required='True', - default='alexnet') - - parser.add_argument('-d', '--dataset', - help='CIFAR10, CIFAR100, or IMAGENET', - required='True', - default='IMAGENET') - - parser.add_argument('-f', '--format', - help='Data format: fp32, fp16, bfloat16, fixedpt', - required='True', - default='fp32') - - parser.add_argument('-P', '--precision', - help='FP32 or FP16', - default='FP16') - - parser.add_argument('-o', '--output', - help='output path', - default='../output/') - - parser.add_argument('-C', '--cuda', - help='True or False', - type=str2bool, - nargs='?', - const=True, - default=True) - - parser.add_argument('-i', '--injections', - help='The number of injections to perform', - type=int, - default=-1) - - parser.add_argument('-I', '--injectionLocation', - help='Injection Location (0-5)', - type=int, - default=-0) - - parser.add_argument('-R', '--radix', - help='Radix point for number format, from LSB', - type=int, - default=-1) - - parser.add_argument('-B', '--bitwidth', - help='Bitwidth of number format', - type=int, - default=32) - - parser.add_argument('-a', '--bias', - help='Bias value for AdaptivFloat number format', - type=int, - default=None) - - parser.add_argument('-r', '--training', - help='When enabled, this is training data. When disabled, this is testing data', - type=str2bool, - nargs='?', - const=True, - default=False) - - parser.add_argument('-w', '--workers', - help='Number of workers for dataloader', - type=int, - default=0) - - parser.add_argument('-q', '--quantize', - help='Enable neuron quantization (def=8 bits)', - type=str2bool, - nargs='?', - const=True, - default=False) - - parser.add_argument('-e', '--errormodel', - help='Single bit flip error model during quantization. -q needs to be enabled too', - type=str2bool, - nargs='?', - const=True, - default=False) - - parser.add_argument('-v', '--verbose', - help='True or False', - type=str2bool, - nargs='?', - const=True, - default=False) - - parser.add_argument('-D', '--debug', - help='True or False', - type=str2bool, - nargs='?', - const=True, - default=False) + + parser.add_argument('-b', '--batchsize', help='Batch Size for inference', type = int, required='True', default=8) + parser.add_argument('-n', '--dnn', help='Neural network architecture backbone', required = 'True', default='alexnet') + parser.add_argument('-d', '--dataset', help='CIFAR10, CIFAR100, IMAGENET or, COCO', required='True', default='IMAGENET') + parser.add_argument('-f', '--format', help='Data format: fp32, fp16, bfloat16, fixedpt', required='True', default='fp32') + parser.add_argument('-P', '--precision', help='FP32 or FP16', default='FP16') + parser.add_argument('-o', '--output', help='output path', default='../output/') + parser.add_argument('-C', '--cuda', help='True or False', type=str2bool, nargs='?', const=True, default=True) + parser.add_argument('-i', '--injections', help='The number of injections to perform', type=int, default=-1) + parser.add_argument('-I', '--injectionLocation', help='Injection Location (0-5)', type=int, default=-0) + parser.add_argument('-R', '--radix', help='Radix point for number format, from LSB', type=int, default=-1) + parser.add_argument('-B', '--bitwidth', help='Bitwidth of number format', type=int, default=32) + parser.add_argument('-a', '--bias', help='Bias value for AdaptivFloat number format', type=int, default=None) + parser.add_argument('-r', '--training', help='When enabled, this is training data. When disabled, this is testing data', type=str2bool, nargs='?', const=True, default=False) + parser.add_argument('-w', '--workers', help='Number of workers for dataloader', type=int, default=0) + parser.add_argument('-q', '--quantize', help='Enable neuron quantization (def=8 bits)', type=str2bool, nargs='?', const=True, default=False) + parser.add_argument('-e', '--errormodel', help='Single bit flip error model during quantization. -q needs to be enabled too', type=str2bool, nargs='?', const=True, default=False) + parser.add_argument('-v', '--verbose', help='True or False for logging info', type=str2bool, nargs='?', const=True, default=False) + parser.add_argument('-D', '--debug', help='True or False', type=str2bool, nargs='?', const=True, default=False) results = parser.parse_args(args) @@ -176,7 +98,6 @@ def check_args(args=None): # if singlebitflip_in: # assert (quantize_in) - def getBatchsize(): return batchsize_in def getDNN(): return dnn_in def getDataset(): return dataset_in @@ -232,7 +153,10 @@ def str2bool(v): # returns the number of classes for common datasets def getNumClasses(dataset): - if(dataset == 'CIFAR10'): + + if(dataset == 'COCO'): + return 91 + elif(dataset == 'CIFAR10'): return 10 elif(dataset == 'CIFAR100'): return 100 @@ -292,6 +216,44 @@ def getNetwork(networkName, DATASET): elif networkName == "cifar10_nn_v2": MODEL = cifar10_nn.v2(pretrained=True, output_size=getNumClasses(DATASET)) + elif DATASET == "COCO": + + if networkName =='maskrcnn': + MODEL = torchvision.models.detection.maskrcnn_resnet50_fpn(progress=True, num_classes=getNumClasses(DATASET)) + + elif networkName =='frcnn': + MODEL = torchvision.models.detection.fasterrcnn_resnet50_fpn(progress=True, num_classes=getNumClasses(DATASET)) + + elif networkName =='keypointrcnn': + MODEL = torchvision.models.detection.keypointrcnn_resnet50_fpn(progress=True, num_classes=getNumClasses(DATASET)) + + elif networkName =='ssd300': + MODEL = torchvision.models.detection.ssd300_vgg16(progress=True, num_classes=getNumClasses(DATASET)) + + elif networkName =='ssd': + MODEL = torchvision.models.detection.SSD( progress=True, num_classes=getNumClasses(DATASET)) + + elif networkName =='fcos': + MODEL = torchvision.models.detection.fcos_resnet50_fpn(progress=True, num_classes=getNumClasses(DATASET)) + + elif networkName =='ssdlite': + MODEL = torchvision.models.detection.ssdlite320_mobilenet_v3_large( progress=True, num_classes=getNumClasses(DATASET)) + + elif networkName =='retinanet_resnet50_fpn': + MODEL = torchvision.models.detection.retinanet_resnet50_fpn(progress=True, num_classes=getNumClasses(DATASET), pretrained_backbone=True) + + elif networkName =='yolov3s': + MODEL = torch.hub.load('ultralytics/yolov3', 'yolov3s',force_reload=True) + + elif networkName =='yolov5s': + MODEL = torch.hub.load('ultralytics/yolov5', 'yolov5s',force_reload=True) + + elif networkName =='yolov8s': + MODEL = torch.hub.load('ultralytics/yolov8', 'yolov8s',force_reload=True) + + elif networkName =='yolovnas': + from ultralytics import NAS + MODEL = NAS('yolo_nas_s.pt') # Error else: sys.exit("Network does not exist") @@ -314,14 +276,36 @@ def load_dataset(DATASET, BATCH_SIZE, workers=0, training=False, shuffleIn=False ) if include_id: - testset = IdCifar10(root='./data', train=training, download=True, transform=transform) + testset = IdCifar10(root='../data', train=training, download=True, transform=transform) else: - testset = datasets.CIFAR10(root='./data', train=training, download=True, transform=transform) + testset = datasets.CIFAR10(root='../data', train=training, download=True, transform=transform) test_loader = torch.utils.data.DataLoader(testset, batch_size=BATCH_SIZE, shuffle=shuffleIn, num_workers=workers, pin_memory=True) dataiter = iter(test_loader) + + elif DATASET == 'COCO': + # coco_dataset = dataloader.COCODataset(root="../data/extras/train", + # annotation="../data/extras/my_train_coco.json", + # transforms=dataloader.get_transform()) + + coco_dataset = dataloader.COCODataset(root="../data/mscoco/val2017", + annotation="../data/mscoco/annotations/instances_val2017.json", + transforms=dataloader.get_transform()) + + data_loader = torch.utils.data.DataLoader( + coco_dataset, + batch_size = BATCH_SIZE, + shuffle = shuffleIn, + num_workers = workers, + pin_memory=True, + pin_memory_device='cuda', + collate_fn = dataloader.collate_fn, + ) + dataiter = iter(data_loader) + + elif DATASET == 'CIFAR100': transform = transforms.Compose( [ @@ -331,9 +315,9 @@ def load_dataset(DATASET, BATCH_SIZE, workers=0, training=False, shuffleIn=False ) if include_id: - testset = IdCifar100(root='./data', train=training, download=True, transform=transform) + testset = IdCifar100(root='../data', train=training, download=True, transform=transform) else: - testset = datasets.CIFAR100(root='./data', train=training, download=True, transform=transform) + testset = datasets.CIFAR100(root='../data', train=training, download=True, transform=transform) test_loader = torch.utils.data.DataLoader(testset, batch_size=BATCH_SIZE, shuffle=shuffleIn, num_workers=workers, pin_memory=True) @@ -364,8 +348,8 @@ def load_dataset(DATASET, BATCH_SIZE, workers=0, training=False, shuffleIn=False # total_data refers to the total size of the data_loader, for all images desired -def load_custom_dataset(NETWORK, DATASET, BATCH_SIZE, good_images, total_data, - workers = 0, random=True, replacement=True, single=False, singleIndex=0): +def load_custom_dataset(NETWORK, DATASET, BATCH_SIZE, good_images, total_data, workers = 0, random=True, replacement=True, single=False, singleIndex=0): + if random: if replacement: if single == False: @@ -386,7 +370,7 @@ def load_custom_dataset(NETWORK, DATASET, BATCH_SIZE, good_images, total_data, ] ) - testset = IdCifar10(root='./data', train=False, + testset = IdCifar10(root='../data', train=False, download=True, transform=transform) test_loader = torch.utils.data.DataLoader(testset, batch_size=BATCH_SIZE, sampler=custom_sampler, num_workers=workers, pin_memory=True) @@ -400,7 +384,7 @@ def load_custom_dataset(NETWORK, DATASET, BATCH_SIZE, good_images, total_data, ] ) - testset = IdCifar100(root='./data', train=False, + testset = IdCifar100(root='../data', train=False, download=True, transform=transform) test_loader = torch.utils.data.DataLoader(testset, batch_size=BATCH_SIZE, sampler=custom_sampler, num_workers=workers, pin_memory=True) @@ -421,6 +405,21 @@ def load_custom_dataset(NETWORK, DATASET, BATCH_SIZE, good_images, total_data, num_workers = workers, sampler=custom_sampler, pin_memory=True) dataiter = iter(val_loader) + if DATASET == 'COCO': + coco_data = dataloader.COCODataset(root="../data/mscoco/val2017", + annotation="../data/mscoco/annotations/instances_val2017.json", + transforms=dataloader.get_transform()) + + data_loader = torch.utils.data.DataLoader( + coco_data, + batch_size = BATCH_SIZE, + num_workers = workers, + sampler=custom_sampler, + pin_memory=True, + pin_memory_device='cuda', + collate_fn = dataloader.collate_fn, + ) + dataiter = iter(data_loader) return dataiter class IdCifar10(datasets.CIFAR10): @@ -444,6 +443,7 @@ def __init__(self, root, train=False, self.data = [] self.targets = [] self.img_names = [] + # now load the picked numpy arrays for file_name, checksum in downloaded_list: file_path = os.path.join(self.root, self.base_folder, file_name) @@ -552,7 +552,6 @@ def get_custom_sampler_full(indices): return sampler - ################################################################# ############## HELPER METHODS FOR IMG PROCESSING ############### ################################################################# @@ -580,7 +579,6 @@ def softmax(x): def diff_top2(data): return (data[0] - data[1]).item() * 100 - ################################################################# ##################### HELPER METHODS FOR I/O #################### ################################################################# @@ -593,7 +591,6 @@ def save_data(path, file_name, data, compress = True): f.close() - def load_file(file_name, compress = True): f = bz2.BZ2File(file_name + '.p.bz2', "rb") if compress else open(file_name.strip('.p.bz2'),"rb") fileIn= cPickle.load(f) @@ -627,4 +624,3 @@ def getNumSysName(name, bits=16, radix_up=5, radix_down=10, bias=None): else: sys.exit("Number format not supported") - diff --git a/val/__init__.py b/val/__init__.py old mode 100644 new mode 100755 diff --git a/val/test_num_sys.py b/val/test_num_sys.py old mode 100644 new mode 100755