192
edits
Difference between revisions of "Gpubox"
Jump to navigation
Jump to search
→Specific VM Configurations
(pre tags instead of `, and lots of edits) |
|||
| (10 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
= Hosted Services = | |||
= gpubox Setup | {| class="wikitable" | ||
|+ On gpubox | |||
|- | |||
! Hostname:Port !! Description | |||
|- | |||
| [https://gpubox.local:8006/ gpubox.local:8006] || Proxmox admin | |||
|- | |||
| [http://dockerhost.local:3000/ dockerhost.local:3000] || Open WebUI (to play with LLMs) | |||
|- | |||
| [https://ipmi-compute-2-171.local/ ipmi-compute-2-171.local] || IPMI | |||
|} | |||
= gpubox Setup = | |||
== Bare Metal Configuration == | == Bare Metal Configuration == | ||
| Line 102: | Line 114: | ||
* Upload <pre>debian-13.3.0-amd64-netinst.iso</pre> to storage through the proxmox web ui | * Upload <pre>debian-13.3.0-amd64-netinst.iso</pre> to storage through the proxmox web ui | ||
* Create a minimal Debian 13 template | * Create a minimal Debian 13 template | ||
** <pre>apt install -y ufw fail2ban curl git zsh sudo</pre> | ** <pre>apt install -y ufw fail2ban curl git zsh sudo net-tools</pre> | ||
** <pre>sudo apt update && sudo apt full-upgrade -y</pre> | ** <pre>sudo apt update && sudo apt full-upgrade -y</pre> | ||
* Make a user called <pre>deb</pre> with sudo | * Make a user called <pre>deb</pre> with sudo | ||
| Line 116: | Line 128: | ||
=== ollama-2080 === | === ollama-2080 === | ||
* Install | * Install nvidia drivers https://www.xda-developers.com/nvidia-stopped-supporting-my-gpu-so-i-started-self-hosting-llms-with-it/ | ||
* Use ollama to pull and run | ** Pin the driver version so you don't have to re-run the nvidia installer every time the kernel gets updated | ||
* Install ollama with <pre>curl -fsSL https://ollama.com/install.sh | sh</pre> | |||
* Use ollama to pull and run deepseek-r1:8b | |||
* <pre>sudo ufw allow from 10.0.0.0/24 to any port 11434 proto tcp</pre> | |||
* Verify: http://ollama.local:11434/ should show the message <pre>Ollama is running.</pre> | * Verify: http://ollama.local:11434/ should show the message <pre>Ollama is running.</pre> | ||
=== imgtotext === | |||
* Install ollama as above | |||
* <pre>ollama run hf.co/noctrex/ZwZ-8B-GGUF:Q8_0</pre> from the page https://huggingface.co/noctrex/ZwZ-8B-GGUF (i pressed the image-to-text tag and looked at trending models) | |||
* http://imgtotext.local:11434/ should show ollama is running | |||
=== dockerhost === | === dockerhost === | ||
| Line 127: | Line 147: | ||
``` | ``` | ||
* Add user <pre>docker</pre> to do docker stuff. Do NOT give <pre>docker</pre> sudo. | * Add user <pre>docker</pre> to do docker stuff. Do NOT give <pre>docker</pre> sudo. | ||
==== Install openwebui ==== | |||
* As the docker user, make directories <pre>~/git/openwebui</pre> | * As the docker user, make directories <pre>~/git/openwebui</pre> | ||
* Make a docker compose file at <pre>~/git/openwebui/docker-compose.yaml</pre> | * Make a docker compose file at <pre>~/git/openwebui/docker-compose.yaml</pre> | ||
** <pre>services: | |||
open-webui: | |||
build: | |||
context: . | |||
dockerfile: Dockerfile | |||
image: ghcr.io/open-webui/open-webui:${WEBUI_DOCKER_TAG-main} | |||
container_name: open-webui | |||
volumes: | |||
- open-webui:/app/backend/data | |||
ports: | |||
- ${OPEN_WEBUI_PORT-3000}:8080 | |||
environment: | |||
- 'OLLAMA_BASE_URL=http://ollama-2080.local:11434' | |||
- 'WEBUI_SECRET_KEY=secretkeyhere' | |||
extra_hosts: | |||
- host.docker.internal:host-gateway | |||
restart: unless-stopped | |||
volumes: | |||
open-webui: {} | |||
</pre> | |||
** Eventually, we'll check this in to git. | |||
* In <pre>~/git/openwebui</pre>, run <pre>docker compose up</pre> | * In <pre>~/git/openwebui</pre>, run <pre>docker compose up</pre> | ||
** Note: newer docker uses <pre>docker compose</pre>, not <pre>docker-compose</pre> | ** Note: newer docker uses <pre>docker compose</pre>, not <pre>docker-compose</pre> | ||
* I had to do some hole-punching in ufw to get open-webui to see ollama2080 | |||
* Useful commands | * Useful commands | ||
** <pre>sudo ss -plnt # | ** <pre>sudo ss -plnt # Lists ports this machine is listening on | ||
ip -4 a # Get this machine's IP address on the local network | ip -4 a # Get this machine's IP address on the local network | ||
</pre> | </pre> | ||
| Line 138: | Line 183: | ||
=== ai-conductor === | === ai-conductor === | ||
* TBD | * TBD | ||
=== if you're using a 1080 Ti or 1080 === | |||
<pre>sudo apt purge "*nvidia*" | |||
sudo apt autoremove --purge | |||
</pre> | |||
then reboot. | |||
== Key Commands == | == Key Commands == | ||