4. Skip to content

Privacy Fingerprint

Running inference

Initializing search

privfp-experiments

Home
Background
Setup
Open LLMs (Generate)
Open Extraction (Structure)
Standardisation (Structure)
(Py)CorrectMatch (Scorer)
SHAP (Explainer)

This is an experimental repository and is a work in progress.

Privacy Fingerprint

privfp-experiments

Home
Background
Background
Setup
Setup
Open LLMs (Generate)
Open LLMs (Generate)
Open Extraction (Structure)
Open Extraction (Structure)
- Open-source Named-Entity Recognition Models
- De-identification Methods
Standardisation (Structure)
Standardisation (Structure)
- Standardising Entity Types and their Values
- Standardising with Knowledge Graphs
(Py)CorrectMatch (Scorer)
(Py)CorrectMatch (Scorer)
- What is Correct Match?
- How does Correct Match Work?
SHAP (Explainer)
SHAP (Explainer)

4. Running inference

How do we run inference on an open source LLM? The following table contains a (non-exhaustive) list of methods to interact with an open source LLM.

Command linePython libraryApplication

Projects

llama.cppOllama

Build and run quantised LLMs via the command line.

Serve and run any GGUF format LLM via the Ollama CLI.

Projects

HF TransformersLangChain

HF Transformers provides APIs and tools to easily run inference on LLMs available from the HF Hub.

A framework for developing applications powered by LLMs.

Projects

HF Text generation inferenceText generation web UILM Studio

A toolkit for deploying and serving LLMs.

A Gradio web UI for LLMs.

An application to discover, download, and run local LLMs.

The Awesome-LLM repository also contains a useful list of tools for deploying LLMs.

Deploying tools

Setting up open source LLMs

Evaluating generated outputs

Made with Material for MkDocs