logo
Browse Source

Update

Signed-off-by: shiyu22 <shiyu.chen@zilliz.com>
training
shiyu22 3 years ago
parent
commit
b8b5965507
  1. 103
      README.md
  2. BIN
      pic/operator.png

103
README.md

@ -1,20 +1,111 @@
# Image Embedding Operator with Resnet50
Authors: name or github-name(email)
Authors: Kyle, shiyu22
## Overview
Introduce the functions of op and the model used.
This Operator generates feature vectors from the pytorch pretrained **Resnet50** mode, which is trained on [COCO dataset](https://cocodataset.org/#download).
**Resnet** models were proposed in “[Deep Residual Learning for Image Recognition](https://arxiv.org/abs/1512.03385)”, this model was the winner of ImageNet challenge in 2015. The fundamental breakthrough with ResNet was it allowed us to train extremely deep neural networks with 150+layers successfully. Prior to ResNet training very deep neural networks was difficult due to the problem of vanishing gradients.
## Interface
The interface of all the functions in op. (input & output)
`Class Resnet50ImageEmbedding(Operator)` [source](./resnet50_image_embedding.py)
`__init__(self, model_name: str)`
**param:**
- model_name(str): the model name for embedding, like 'resnet50'.
`__call__(self, img_tensor: torch.Tensor)`
**param:**
- img_tensor(torch.Tensor): the normalized image tensor.
**return:**
- cnn(numpy.ndarray): the embedding of image.
## How to use
- Requirements from requirements.txt
- How it works in some typical pipelines and the yaml example.
### Requirements
You can get the required python package by [requirements.txt](./requirements.txt) and [pytorch/requirements.txt](./pytorch/requirements.txt). In fact, Towhee will automatically install these packages when you first load the Operator Repo, so you don't need to install them manually, here is just a list.
- towhee
- torch
- torchvision
- numpy
### How it works
The towhee/resnet50_image_embedding Operator implements the function of image embedding, which can add to the pipeline, for example, it's the key Operator within [image_embedding_resnet50](https://hub.towhee.io/towhee/image-embedding-resnet50) pipeline, and it is the red box in the picture below.
![img](./pic/operator.png)
When using this Operator to write Pipline's Yaml file, you need to declare the following content according to the interface of resnet50_image_embedding class:
```yaml
operators:
-
name: 'embedding_model'
function: 'towhee/resnet50-image-embedding'
tag: 'main'
init_args:
model_name: 'resnet50'
inputs:
-
df: 'image_preproc'
name: 'img_tensor'
col: 0
outputs:
-
df: 'embedding'
iter_info:
type: map
dataframes:
-
name: 'image_preproc'
columns:
-
name: 'img_transformed'
vtype: 'torch.Tensor'
-
name: 'embedding'
columns:
-
name: 'cnn'
vtype: 'numpy.ndarray'
```
We can see that in yaml, the **operator** part declares the `init_args` of the class and the` input` and `output`dataframe, and the **dataframe** declares the parameter `name` and `vtype`.
### File Structure
Here is the main file structure of the `resnet50-image-embedding` Operator. If you want to learn more about the source code or modify it yourself, you can learn from it.
```bash
├── .gitattributes
├── .gitignore
├── README.md
├── __init__.py
├── requirements.txt #General python dependency package
├── resnet50_image_embedding.py #The python file for Towhee, it defines the interface of the system and usually does not need to be modified.
├── resnet50_image_embedding.yaml #The YAML file contains Operator information, such as model frame, input, and output.
├── pytorch #The directory of the pytorh
│   ├── __init__.py
│   ├── model #The directory of the pytorch model, which can store data such as weights.
│   ├── requirements.txt #The python dependency package for the pytorch model.
│   └── model.py #The code of the pytorch model, including the initialization model and prediction.
├── test_data/ #The directory of test data, including test.jpg
└── test_resnet50_image_embedding.py #The unittest file of the Operator.
```
## Reference
Model paper link.
- https://pytorch.org/hub/pytorch_vision_resnet/
- https://arxiv.org/abs/1512.03385

BIN
pic/operator.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 267 KiB

Loading…
Cancel
Save