A tutorial details how to expose a text classification pipeline based on a large language model (LLM) running locally as a REST API. The solution combines three open-source tools: Scikit-LLM, FastAPI, and Ollama, using the llama3 model.
Scikit-LLM allows integrating an LLM into pipelines compatible with scikit-learn, the reference machine learning library in Python. The use case presented is ‘zero-shot’ classification, meaning a pre-trained model is transformed into a classifier using only the list of possible categories, without additional training. According to the author, putting this type of pipeline into production often runs into paid cloud APIs and latency spikes.
To avoid these constraints, the configuration redirects Scikit-LLM to a local Ollama server, at the address http://localhost:11434/v1/, instead of a commercial API. Ollama must be installed and the llama3 model downloaded locally; the author refers to the official Ollama website for this. The tutorial was done with Visual Studio Code and Python 3.11.
Training, Serializing, and Serving the Classifier
The first file, train.py, builds a ZeroShotGPTClassifier associated with the custom_url::llama3 model, then fits it on three example texts, with the labels ‘Battery Issue’, ‘Positive Feedback’, and ‘Hardware Damage’. The pipeline is then saved to a file named scikit_llm_model.joblib using the joblib library.
The second file, app.py, reloads the configuration, loads the pipeline on startup, and defines a /predict endpoint with FastAPI. This endpoint accepts multiple texts in a single call and returns a prediction for each. The blocking processing of Scikit-LLM is offloaded to a separate thread, so as not to freeze the asynchronous server. The server is launched with uvicorn on 127.0.0.1:8000.
Once the server is started, the interactive interface is accessible at http://127.0.0.1:8000/docs. In the provided example, three reviews about a phone are submitted: ‘My phone dies by noon even if I don’t use it.’, ‘I dropped it on the sidewalk and the glass shattered.’, and ‘This is the best device I have ever purchased!’. The returned predictions are, in order, ‘Battery Issue’, ‘Hardware Damage’, and ‘Positive Feedback’.
The Model’s Limitations
The author reminds us that the LLM’s weights are not modified: the labels passed to the fit() method are simply stored and presented to the model during inference, akin to a multiple-choice question in a prompt. The classifier is thus constrained to assign each text to one of the labels in the list, regardless of the content submitted.
According to the author, this constraint provides greater predictability for the LLM, at the cost of a ‘viewpoint’ limited to the defined categories. Finally, thanks to Scikit-LLM’s compatibility with scikit-learn objects and syntax, the pipeline could be enhanced with additional preprocessing steps.
