LLM plugin for models hosted by Anyscale Endpoints
Project description
llm-anyscale-endpoints
LLM plugin for models hosted by Anyscale Endpoints
Installation
First, install the LLM command-line utility.
Now install this plugin in the same environment as LLM.
llm install llm-anyscale-endpoints
Configuration
You will need an API key from Anyscale Endpoints. You can obtain one here.
You can set that as an environment variable called LLM_ANYSCALE_ENDPOINTS_KEY
, or add it to the llm
set of saved keys using:
llm keys set anyscale-endpoints
Enter key: <paste key here>
Usage
To list available models, run:
llm models list
You should see a list that looks something like this:
AnyscaleEndpoints: meta-llama/Llama-2-7b-chat-hf
AnyscaleEndpoints: meta-llama/Llama-2-13b-chat-hf
AnyscaleEndpoints: meta-llama/Llama-2-70b-chat-hf
To run a prompt against a model, pass its full model ID to the -m
option, like this:
llm -m meta-llama/Llama-2-70b-chat-hf \
'Five strident names for a pet walrus' \
--system 'You love coming up with creative names for pets'
You can set a shorter alias for a model using the llm aliases
command like so:
llm aliases set llama70b meta-llama/Llama-2-70b-chat-hf
Now you can prompt Claude using:
cat llm_anyscale_endpoints.py | llm -m llama70b -s 'explain this code'
Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd llm-anyscale-endpoints
python3 -m venv venv
source venv/bin/activate
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for llm-anyscale-endpoints-0.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6fdfbef1d9deebc17f19ce1311bcb7212f45f63282f4fa06804a8bc972fef54d |
|
MD5 | d2b98b7ca9c62f2a96eb3ed97a157fcf |
|
BLAKE2b-256 | 9f01d3613b61a533a1a4f5d510e182bde539735abf5b49fe39bf1708b823851d |
Hashes for llm_anyscale_endpoints-0.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 255cf363c461aaaffe93231670a0953a39c8f22d3070b399f7b0c0e1e14cf95a |
|
MD5 | 3198104a3768e133de5ce0e6e673c829 |
|
BLAKE2b-256 | e12093403097df739c330383ebf25844b2e3fa77462e6faaaae7aa227c59859a |