Tensorflow Model Deployment and Inferencing with Kubeflow

In the last part of this series, we trained a Tensorflow model to classify the images of cats and dogs. The model is stored in a shared Kubernetes persistent volume claim (PVC) which can be accessed by another Kubeflow Notebook Server to test the model.

Remember, this series aims not to build an extremely complex neural network but to demonstrate how Kubeflow helps organizations with machine learning operations (MLOps).

👁 Image

Launch a new CPU-based Jupyter Notebook Server and upload the notebook available on GitHub. This notebook validates the model by passing a few images.

👁 Image

Follow the same steps to launch the Notebook Server based on the image, janakiramm/infer. Make sure you mount the shared PVC – models.

This notebook loads the TensorFlow model and performs the classification based on sample images.

👁 Image

The infer function accepts a file and returns the prediction.

👁 Image

👁 Image

👁 Image

Let’s now deploy the model in TensorFlow Serving running in Kubernetes. Start by cloning the Github repository that has everything we need to run the inference code.

git clone https://github.com/janakiramm/kubeflow-notebook-tutorial.git

Navigate to the inference directory to find the YAML files and other related assets.

Let’s deploy TensorFlow Serving in the kubeflow-user-example-com namespace and expose it as a NodePort service. It’s the same namespace where the Jupyter Notebook Servers are running.

cd inference
kubectl apply -f tf-serve-deploy.yaml
kubectl apply -f tf-serve-service.yaml

👁 Image

Below are YAML specifications for the TF Serving deployment and service.

apiVersion: apps/v1
kind: Deployment
metadata:
 labels:
 app: dogs-vs-cats
 name: dogs-vs-cats-v1
 namespace: kubeflow-user-example-com
spec:
 selector:
 matchLabels:
 app: dogs-vs-cats
 template:
 metadata:
 labels:
 app: dogs-vs-cats
 version: v1
 spec:
 containers:
 - args:
 - --port=9000
 - --rest_api_port=8500
 - --model_name=dogs-vs-cats
 - --model_base_path=/models
 command:
 - /usr/bin/tensorflow_model_server
 image: tensorflow/serving:latest
 imagePullPolicy: IfNotPresent
 livenessProbe:
 initialDelaySeconds: 30
 periodSeconds: 30
 tcpSocket:
 port: 9000
 name: dogs-vs-cats
 ports:
 - containerPort: 9000
 - containerPort: 8500
 volumeMounts:
 - mountPath: /models
 name: model-serve-storage
 volumes:
 - name: model-serve-storage
 persistentVolumeClaim:
 claimName: models

apiVersion: v1
kind: Service
metadata:
 labels:
 app: dogs-vs-cats
 name: dogs-vs-cats-service
 namespace: kubeflow-user-example-com
spec:
 ports:
 - name: http-tf-serving
 port: 8500
 targetPort: 8500
 nodePort: 31000
 - name: grpc-tf-serving
 port: 9000
 targetPort: 9000
 nodePort: 31001 
 selector:
 app: dogs-vs-cats
 type: NodePort

We are essentially mounting the same PVC used by the Jupyter Notebook Servers to serve the model.

The TF Serving endpoint is available as a NodePort on the Kubeflow cluster.

Since Kubeflow relies on Istio for authorizing requests, we need to apply an authorization policy to allow requests to TF Serving.

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: default
 namespace: kubeflow-user-example-com
spec:
 rules:
 - to:
 - operation:
 methods: ["GET","POST"]
 paths: ["/v1/models/*"]

kubectl apply -f tf-serve-auth.yaml

It’s time to invoke the endpoint from a Python Client. Let’s create a virtual environment and install the required modules.

python3 -m venv inferenv
source inferenv/bin/activate

pip install -r requirements.txt

Below is the Python client code we use for inference.

import argparse
import json

import numpy as np
import requests
import tensorflow
import PIL
from tensorflow.keras.preprocessing import image

ap = argparse.ArgumentParser()
ap.add_argument("-i", "--image", required=True,
 help="path of the image")
ap.add_argument("-u", "--uri", required=True,
 help="URI of model server")

args = vars(ap.parse_args())

image_path = args['image']
uri = args['uri']

img = image.img_to_array(image.load_img(image_path, target_size=(128, 128))) / 255.

payload = {
 "instances": [{'conv2d_3_input': img.tolist()}]
}

r = requests.post(uri+'/v1/models/dogs-vs-cats:predict', json=payload)
pred = json.loads(r.content.decode('utf-8'))
predict=np.asarray(pred['predictions']).argmax(axis=1)[0]
print( "Dog" if predict==1 else "Cat" )

Let’s run the Python client by passing the TF Serving URL and a sample image. When sending sample1.jpg, we see the prediction as a dog and a cat when using sample2.jpg.

👁 Image

HOST=http://10.0.0.54:31000
python infer.py -i sample1.jpg -u $HOST

Replace HOST with an appropriate IP and port-based on your cluster and the TF Serving NodePort service.

👁 Image

HOST=http://10.0.0.54:31000
python infer.py -i sample2.jpg -u $HOST

👁 Image

As you can see, the classification is accurate for the images that we sent.

This concludes the series on Kubeflow Jupyter Notebook Servers where we explored the end-to-end MLOps scenario of configuring the environment, performing data preparation, training, deployment, and inference.