Creating new TensorMetric (MultilabelAccuracy), running into _orig_call error

implementation help
1

I’m trying to implement a new TensorMetric, but I’m running into the error: ModuleAttributeError: 'MultilabelAccuracy' object has no attribute '_orig_call'

Sample code:

import torch
from pytorch_lightning.metrics.metric import TensorMetric

num_classes = 3
target = torch.randint(0, 2, size=(10, num_classes))
pred = torch.rand(10, num_classes)

# Page 44:
# https://users.ics.aalto.fi/jesse/talks/Multilabel-Part01.pdf
class MultilabelAccuracy(TensorMetric):
    def __init__(self, threshold: float = 0.5):
        super().__init__('multilabel-accuracy')
        self.threshold = torch.tensor(threshold)
    
    def forward(self, pred, target):
        if pred.is_floating_point():
            # scale between 0 and 1
            pred = pred.sigmoid()
            # print("pred_sig, target", list(zip(pred, target)))
            # cast to 0/1
            pred = (pred.data >= self.threshold).long()
        
        tensor_and = torch.logical_and(pred, target)
        tensor_or = torch.logical_or(pred, target)
        # floating scaler division
        tensor_div = torch.true_divide(tensor_and.sum(dim=1), tensor_or.sum(dim=1))
        # print("tensor_div:", tensor_div)
        # sum all and then divide by i rows == mean
        return tensor_div.mean()
    
ml_acc = MultilabelAccuracy(threshold=0.5)
ml_acc(pred, target)

ModuleAttributeError

ModuleAttributeError                      Traceback (most recent call last)
<ipython-input-57-a13828eef952> in <module>
----> 1 ml_acc(pred, target)

~/anaconda3/envs/pytorchlit/lib/python3.7/site-packages/pytorch_lightning/metrics/metric.py in __call__(self, *args, **kwargs)
     82             return x.to(device=self.device, dtype=self.dtype, non_blocking=True)
     83 
---> 84         return apply_to_collection(self._orig_call(*args, **kwargs), torch.Tensor,
     85                                    _to_device_dtype)
     86 

~/anaconda3/envs/pytorchlit/lib/python3.7/site-packages/torch/nn/modules/module.py in __getattr__(self, name)
    770                 return modules[name]
    771         raise ModuleAttributeError("'{}' object has no attribute '{}'".format(
--> 772             type(self).__name__, name))
    773 
    774     def __setattr__(self, name: str, value: Union[Tensor, 'Module']) -> None:

ModuleAttributeError: 'MultilabelAccuracy' object has no attribute '_orig_call'

Related

Might be related to this issue?: validation_epoch_end needs to return CUDA tensors · Issue #2442 · Lightning-AI/lightning · GitHub

Question

What am I’m doing wrong / missing?

Environment (conda)

* CUDA:
	- GPU:
		- GeForce GTX 1080 Ti
	- available:         True
	- version:           10.2
* Packages:
	- numpy:             1.19.1
	- pyTorch_debug:     False
	- pyTorch_version:   1.6.0
	- pytorch-lightning: 0.9.0
	- tensorboard:       2.2.0
	- tqdm:              4.48.2
* System:
	- OS:                Linux
	- architecture:
		- 64bit
		- 
	- processor:         x86_64
	- python:            3.7.7
	- version:           #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020
2

Oops, you forgot to call super().__init__('your-metric-name') :grinning:

Here it is working on Google Colab.

The snippet:

import torch
from pytorch_lightning.metrics.metric import TensorMetric


num_classes = 3
target = torch.randint(0, 2, size=(10, num_classes))
pred = torch.rand(10, num_classes)

# Page 44:
# https://users.ics.aalto.fi/jesse/talks/Multilabel-Part01.pdf
class MultilabelAccuracy(TensorMetric):
    def __init__(self, threshold: float = 0.5):
        super().__init__('cool-metric')
        self.threshold = torch.tensor(threshold)
    
    def forward(self, pred, target):
        # scale between 0 and 1
        pred = pred.sigmoid()
        # cast to 0/1
        pred_bool = (pred.data >= self.threshold).long()
        vector_and = torch.logical_and(pred, target)
        vector_or = torch.logical_or(pred, target)
        # floating scaler division
        vector_div = torch.true_divide(vector_and.sum(dim=1), vector_or.sum(dim=1))
        # sum all and then divide by i rows == mean
        return vector_div.mean()
    
ml_acc = MultilabelAccuracy(threshold=0.5)
ml_acc(pred, target)  # >>> tensor(0.3000)
1 Like
3

Thank you! I should have realized that about the super call.

I just copied the example code from the documentation and started from there, but that one didn’t have a init call.

1 Like
4

Fixed the code in the original post (wrongly used pred instead of pred_bool). Also, added a check whether the prediction was already in binary format.

5

One more thing to note :slight_smile: :

You should register self.threshold as a buffer to make sure all the device and dtype changes are applied correctly:

class MultilabelAccuracy(TensorMetric):
    def __init__(self, threshold: float = 0.5):
        super().__init__('cool-metric')
        self.register_buffer('threshold', torch.tensor(threshold))
    
    def forward(self, pred, target):
        # scale between 0 and 1
        pred = pred.sigmoid()
        # cast to 0/1
        pred_bool = (pred.data >= self.threshold).long()
        vector_and = torch.logical_and(pred, target)
        vector_or = torch.logical_or(pred, target)
        # floating scaler division
        vector_div = torch.true_divide(vector_and.sum(dim=1), vector_or.sum(dim=1))
        # sum all and then divide by i rows == mean
        return vector_div.mean()
1 Like