towhee
/
resnet-image-embedding
copied
2 changed files with 97 additions and 6 deletions
@ -1,20 +1,111 @@ |
|||||
# Image Embedding Operator with Resnet50 |
# Image Embedding Operator with Resnet50 |
||||
|
|
||||
Authors: name or github-name(email) |
|
||||
|
Authors: Kyle, shiyu22 |
||||
|
|
||||
## Overview |
## Overview |
||||
|
|
||||
Introduce the functions of op and the model used. |
|
||||
|
This Operator generates feature vectors from the pytorch pretrained **Resnet50** mode, which is trained on [COCO dataset](https://cocodataset.org/#download). |
||||
|
|
||||
|
**Resnet** models were proposed in “[Deep Residual Learning for Image Recognition](https://arxiv.org/abs/1512.03385)”, this model was the winner of ImageNet challenge in 2015. The fundamental breakthrough with ResNet was it allowed us to train extremely deep neural networks with 150+layers successfully. Prior to ResNet training very deep neural networks was difficult due to the problem of vanishing gradients. |
||||
|
|
||||
## Interface |
## Interface |
||||
|
|
||||
The interface of all the functions in op. (input & output) |
|
||||
|
`Class Resnet50ImageEmbedding(Operator)` [source](./resnet50_image_embedding.py) |
||||
|
|
||||
|
`__init__(self, model_name: str)` |
||||
|
|
||||
|
**param:** |
||||
|
|
||||
|
- model_name(str): the model name for embedding, like 'resnet50'. |
||||
|
|
||||
|
`__call__(self, img_tensor: torch.Tensor)` |
||||
|
|
||||
|
**param:** |
||||
|
|
||||
|
- img_tensor(torch.Tensor): the normalized image tensor. |
||||
|
|
||||
|
**return:** |
||||
|
|
||||
|
- cnn(numpy.ndarray): the embedding of image. |
||||
|
|
||||
## How to use |
## How to use |
||||
|
|
||||
- Requirements from requirements.txt |
|
||||
- How it works in some typical pipelines and the yaml example. |
|
||||
|
### Requirements |
||||
|
|
||||
|
You can get the required python package by [requirements.txt](./requirements.txt) and [pytorch/requirements.txt](./pytorch/requirements.txt). In fact, Towhee will automatically install these packages when you first load the Operator Repo, so you don't need to install them manually, here is just a list. |
||||
|
|
||||
|
- towhee |
||||
|
- torch |
||||
|
- torchvision |
||||
|
- numpy |
||||
|
|
||||
|
### How it works |
||||
|
|
||||
|
The towhee/resnet50_image_embedding Operator implements the function of image embedding, which can add to the pipeline, for example, it's the key Operator within [image_embedding_resnet50](https://hub.towhee.io/towhee/image-embedding-resnet50) pipeline, and it is the red box in the picture below. |
||||
|
|
||||
|
 |
||||
|
|
||||
|
When using this Operator to write Pipline's Yaml file, you need to declare the following content according to the interface of resnet50_image_embedding class: |
||||
|
|
||||
|
```yaml |
||||
|
operators: |
||||
|
- |
||||
|
name: 'embedding_model' |
||||
|
function: 'towhee/resnet50-image-embedding' |
||||
|
tag: 'main' |
||||
|
init_args: |
||||
|
model_name: 'resnet50' |
||||
|
inputs: |
||||
|
- |
||||
|
df: 'image_preproc' |
||||
|
name: 'img_tensor' |
||||
|
col: 0 |
||||
|
outputs: |
||||
|
- |
||||
|
df: 'embedding' |
||||
|
iter_info: |
||||
|
type: map |
||||
|
dataframes: |
||||
|
- |
||||
|
name: 'image_preproc' |
||||
|
columns: |
||||
|
- |
||||
|
name: 'img_transformed' |
||||
|
vtype: 'torch.Tensor' |
||||
|
- |
||||
|
name: 'embedding' |
||||
|
columns: |
||||
|
- |
||||
|
name: 'cnn' |
||||
|
vtype: 'numpy.ndarray' |
||||
|
``` |
||||
|
|
||||
|
We can see that in yaml, the **operator** part declares the `init_args` of the class and the` input` and `output`dataframe, and the **dataframe** declares the parameter `name` and `vtype`. |
||||
|
|
||||
|
### File Structure |
||||
|
|
||||
|
Here is the main file structure of the `resnet50-image-embedding` Operator. If you want to learn more about the source code or modify it yourself, you can learn from it. |
||||
|
|
||||
|
```bash |
||||
|
├── .gitattributes |
||||
|
├── .gitignore |
||||
|
├── README.md |
||||
|
├── __init__.py |
||||
|
├── requirements.txt #General python dependency package |
||||
|
├── resnet50_image_embedding.py #The python file for Towhee, it defines the interface of the system and usually does not need to be modified. |
||||
|
├── resnet50_image_embedding.yaml #The YAML file contains Operator information, such as model frame, input, and output. |
||||
|
├── pytorch #The directory of the pytorh |
||||
|
│ ├── __init__.py |
||||
|
│ ├── model #The directory of the pytorch model, which can store data such as weights. |
||||
|
│ ├── requirements.txt #The python dependency package for the pytorch model. |
||||
|
│ └── model.py #The code of the pytorch model, including the initialization model and prediction. |
||||
|
├── test_data/ #The directory of test data, including test.jpg |
||||
|
└── test_resnet50_image_embedding.py #The unittest file of the Operator. |
||||
|
``` |
||||
|
|
||||
## Reference |
## Reference |
||||
|
|
||||
Model paper link. |
|
||||
|
- https://pytorch.org/hub/pytorch_vision_resnet/ |
||||
|
- https://arxiv.org/abs/1512.03385 |
||||
|
|
||||
|
|
||||
|
After Width: | Height: | Size: 267 KiB |
Loading…
Reference in new issue