serving

Through Towhee’s api_service interface, you can quickly convert the defined pipelines into http/gRPC services. Let’s take a simple example to see how to use the api_service module

Define Service

import typing as T
from pydantic import BaseModel
from towhee import AutoPipes, api_service
from towhee.serve.io import JSON, TEXT

# Create a service object
# The / interface will be generated and the string passed in desc will be returned
service = api_service.APIService(desc="Welcome")

# Create sentence_embedding pipeline
stn = AutoPipes.pipeline('sentence_embedding')

# Define a /embedding interface，
# the inputs is TEXT(str)
# the output is ndarray
@service.api(path='/embedding', input_model=TEXT(), output_model=NDARRAY())
def embedding(text: str):
    return stn(text).to_list()

# JSON IO supports defining some complex data structures by using pydantic
class Item(BaseModel):
    url: str
    ids: T.List[int]

@service.api(path='/echo', input_model = JSON(Item), output_model=JSON(Item))
def echo(item: Item):
    return item

HTTP Server

start server

from towhee.serve.http.server import HTTPServer
HTTPServer(service).run('0.0.0.0', 8000)

Access Server

Access by Python code

import requests
from towhee.utils.serializer import from_json

requests.get('http://127.0.0.1:8000').json() 
# Welcome

# The http server converts the ndarray type into a json string,
# It can be converted back through towhee's from_json interface.
from_json(requests.post('http://127.0.0.1:8000/embedding', json='hello world').content)
# ndarray

requests.post('http://127.0.0.1:8000/echo', json={'url': 1, 'ids': [1, 2]}).json()
# {'url': '1', 'ids': [1, 2]}

Access by Curl cmd

curl http://127.0.0.1:8000
# Welcome

curl -X POST -H "Content-Type: application/json" -d "hello, towhee" http://127.0.0.1:8000/embedding
# json_str 

curl -X POST -H "Content-Type: application/json" -d '{"url": "http://towhee.io", "ids": [1,2]}' http://127.0.0.1:8000/echo
# {"url": "http://towhee.io", "ids": [1, 2]}

gRPC Server

Start Server

from towhee.serve.grpc.server import GRPCServer
GRPCServer(service).run('0.0.0.0', 8000)

Access by gRPC Client

from towhee.serve.grpc.client import Client
c = Client('0.0.0.0', 8000)

# response
#  code: int 0 success
#  msg: str
#  content: Any
respones = c('/echo', {'url': 1, 'ids': [1, 2]})
respones = c('/embedding', 'hello')
c.close()

# or
with Client('0.0.0.0', 8000) as c:
	respones = c('/echo', {'url': 1, 'ids': [1, 2]})
	respones = c('/embedding', 'hello')

Start HTTP and gRPC Server

from towhee.serve.http.server import HTTPServer
from towhee.serve.grpc.server import GRPCServer
s = GRPCServer(service)
s.start('0.0.0.0', 8002)
HTTPServer(service).run('0.0.0.0', 8000)

IO

Currently supports TEXT, BYTES, JSON and NDARRAY

TEXT/BYTES

TEXT and BYTES correspond to the str and bytes types in Python respectively.

JSON

The JSON IO of the HTTPServer can support the ndarray type, and the client can use it with the from_json function.

The gRPC Server can only handle some basic types and cannot handle ndarray. You can use the to_json function provided by towhee to convert it into json_str.

Can be used with pydantic

NDARRAY

Returned in Json form in HTTP Server, converted to ndarray by from_json.

gRPC client can return ndarray directly.

Only limited dtypes are supported:

np.object
np.float32
np.float64
np.bool_
np.int32
np.int64
np.uint32
np.uint64