Related Documentation

• quick start guide

  get up and running with gguf loader in just a few minutes

  beginner 5 minutes

  get up and running with gguf loader 2.2.0 in just a few minutes! new to gguf loader? check out the homepage to see what makes it special.

  🚀 step 1: install gguf loader

  follow the installation guide for your platform, or download directly:

  • windows: ggufloader_v2.2.0_cpu.exe (cpu-only) · ggufloader_v2.2.0_gpu.exe (nvidia cuda)
  • linux: ggufloader_v2.2.0_linux_x86_64_cpu

  📥 step 2: get a gguf model

  gguf loader runs any gguf-format model. good starter models on hugging face:

  • small (fast): llama-3-8b-instruct gguf (q4_0) — ~4.7 gb, runs great on cpu
  • medium: mistral-7b-instruct gguf (q4_k_m) — ~4.1 gb
  • larger: phi-3.5-mini-instruct gguf or any qwen/llama gguf you like

  download the .gguf file and remember where you saved it.

  🖥️ step 3: load the model

  1. launch gguf loader.
  2. in the model settings sidebar, choose processing mode — cpu only (default, works everywhere) or gpu accelerated (nvidia cuda).
  3. set a context length (8192 is a safe default for most models; 32768 uses more ram).
  4. click load gguf model and select your .gguf file.
  5. wait for the status to show “model ready!” — the header chip turns green with the model name.

  💬 step 4: chat

  type a message in the input box at the bottom and press enter (shift+enter inserts a newline). responses stream in as chatgpt-style bubbles — your messages on the right (amber), the ai’s on the left.

  use the view → text size menu to adjust bubble font size.

  🤖 step 5 (optional): try agent mode

  click 🤖 agent mode: off in the input area to toggle it on:

  1. pick a workspace folder (defaults to ./agent_workspace).
  2. ask the agent to do file work — e.g. “create a file called hello.py that prints ‘hi’“.
  3. the agent plans tool calls, executes them (read/write/edit/search files inside the workspace only), and reports back with live status updates.

  ✅ next steps

  • user guide — everything the app can do
  • addon development — extend gguf loader with addons

  related documentation

  • installation guide

    complete guide to installing gguf loader on windows, macos, and linux

    beginner 5 minutes

    this guide will help you install gguf loader 2.2.0 on your system. want to see what gguf loader can do first? explore the features on our homepage.

    📋 system requirements

    • os: windows 10/11, linux (x86_64), or macos
    • ram: 4 gb minimum (8 gb+ recommended for larger models)
    • storage: 2 gb free space for the app, plus room for model files
    • gpu: optional — cpu-only works everywhere; nvidia cuda is supported on windows and linux
    • python: not required for the installers (everything is bundled)

    🚀 quick installation

    the prebuilt installers bundle python, pyside6, and llama.cpp — you do not need python installed.

    windows

    1. download ggufloader_v2.2.0_cpu.exe (~69 mb) from the releases page. this is the cpu-only build that works on any pc.
    2. have an nvidia gpu with cuda? grab ggufloader_v2.2.0_gpu.exe (~854 mb) instead for gpu acceleration.
    3. double-click the downloaded file and run it. no installation wizard — the app starts immediately.

    ⚠️ windows smartscreen: the first launch may show “windows protected your pc”. click more info → run anyway. this is normal for unsigned open-source binaries.

    linux (x86_64)

    1. download ggufloader_v2.2.0_linux_x86_64_cpu (~121 mb) from the releases page.
    2. make it executable and run:

       chmod +x ggufloader_v2.2.0_linux_x86_64_cpu
       ./ggufloader_v2.2.0_linux_x86_64_cpu
       

       no installation needed — the app starts directly. everything (python, pyside6, llama.cpp) is bundled.

    macos

    no prebuilt installer is published yet. run from source (below) — everything works on macos.

    run from source (any platform)

    for development, or to run on an unsupported platform:

    git clone https://github.com/ggufloader/gguf-loader.git
    cd gguf-loader
    pip install -r requirements.txt
    python main.py
    

    ✅ verify your installation

    run ggufloader_v2.2.0_cpu.exe --version (windows) or ./ggufloader_v2.2.0_linux_x86_64_cpu --version (linux) — it prints gguf loader version 2.2.0 and exits.

    🔧 next steps

    • quick start guide — load your first model
    • troubleshooting — if something isn’t working

    related documentation

    • get up and running with gguf loader 2.2.0 in just a few minutes! new to gguf loader? [check out the homepage](/) to see what makes it special. ## 🚀 step 1: install gguf loader follow the [installation guide](/docs/installation/) for your platform, or download directly: - **windows**: [ggufloader_v2.2.0_cpu.exe](https://github.com/ggufloader/gguf-loader/releases/download/v2.2.0/ggufloader_v2.2.0_cpu.exe) (cpu-only) · [ggufloader_v2.2.0_gpu.exe](https://github.com/ggufloader/gguf-loader/releases/download/v2.2.0/ggufloader_v2.2.0_gpu.exe) (nvidia cuda) - **linux**: [ggufloader_v2.2.0_linux_x86_64_cpu](https://github.com/ggufloader/gguf-loader/releases/download/v2.2.0/ggufloader_v2.2.0_linux_x86_64_cpu) ## 📥 step 2: get a gguf model gguf loader runs any gguf-format model. good starter models on hugging face: - **small (fast)**: [llama-3-8b-instruct gguf (q4_0)](https://huggingface.co/thebloke/llama-3-8b-instruct-gguf) — ~4.7 gb, runs great on cpu - **medium**: [mistral-7b-instruct gguf (q4_k_m)](https://huggingface.co/thebloke/mistral-7b-instruct-v0.2-gguf) — ~4.1 gb - **larger**: [phi-3.5-mini-instruct gguf](https://huggingface.co/microsoft/phi-3.5-mini-instruct-gguf) or any qwen/llama gguf you like download the `.gguf` file and remember where you saved it. ## 🖥️ step 3: load the model 1. launch gguf loader. 2. in the **model settings** sidebar, choose **processing** mode — `cpu only` (default, works everywhere) or `gpu accelerated` (nvidia cuda). 3. set a **context length** (8192 is a safe default for most models; 32768 uses more ram). 4. click **load gguf model** and select your `.gguf` file. 5. wait for the status to show **"model ready!"** — the header chip turns green with the model name. ## 💬 step 4: chat type a message in the input box at the bottom and press **enter** (shift+enter inserts a newline). responses stream in as chatgpt-style bubbles — your messages on the right (amber), the ai's on the left. use the **view → text size** menu to adjust bubble font size. ## 🤖 step 5 (optional): try agent mode click **🤖 agent mode: off** in the input area to toggle it on: 1. pick a **workspace folder** (defaults to `./agent_workspace`). 2. ask the agent to do file work — e.g. *"create a file called hello.py that prints 'hi'"*. 3. the agent plans tool calls, executes them (read/write/edit/search files inside the workspace only), and reports back with live status updates. ## ✅ next steps - [user guide](/docs/user-guide/) — everything the app can do - [addon development](/docs/addon-development/) — extend gguf loader with addons
    • common issues and their fixes. running the app from a terminal (`python main.py` or the installed binary) shows the console log, which is the fastest way to diagnose most problems. ## 🚀 app won't start ### windows smartscreen blocks the .exe - click **more info → run anyway**. the binaries are unsigned open-source builds; this is expected. ### "llama-cpp-python is required but not installed" - the llama.cpp runtime is missing. in the app's **environment** section click **install missing dependencies**, or from source: `pip install -r requirements.txt`. ### dll load errors on windows (`importerror: dll load failed`) - the bundled `llama_cpp/lib` wasn't found. make sure you're running the full package (not just copying a single file), and check the console log for `added windows dll directory: ...`. ## 🧠 model won't load ### "failed to load model" / unknown format - confirm the file is a real gguf file (`.gguf`). corrupted or partial downloads fail; re-download the file. - check the file has enough disk space and isn't on a network drive with strict permissions. ### out of memory / crashes when loading - reduce **context length** (try 4096 or 8192). - switch **processing** to `cpu only` if gpu mode crashes (vram exhaustion). - use a smaller quantized model (q4_0/q4_k_m instead of q8). ### gpu accelerated is slow or doesn't use the gpu - gpu mode requires a cuda-enabled `llama-cpp-python` build. verify with `scripts/verify_gpu_support.py` or run `scripts/install_gpu_llama.bat`/`.sh` (see the [installation guide](/docs/installation/)). - on linux, make sure the nvidia driver and cuda toolkit are installed. ## 💬 chat problems ### send button is disabled - no model is loaded. load a `.gguf` model first. ### responses are empty or garbled - some models need specific prompt formats. check the model card for a chat template; very small models also produce weak output. - lower `temperature`-style randomness isn't exposed per-chat — try a different preset or model. ### chat is slow - smaller context, cpu-only on fast threads, and a smaller quantized model all help. generation speed is dominated by model size and hardware. ## 🤖 agent mode issues ### "please load a model first" - agent mode needs a loaded model — load one, then toggle agent mode on. ### agent can't find files / "path escapes workspace" - the agent is **sandboxed to the workspace folder**. put files you want it to touch inside the workspace, or choose the right folder with the 📁 button. ### agent stuck on "processing..." - generation is single-threaded; long tool chains take time. watch the status messages for progress. ## 💬 floating chat addon ### button doesn't float above other apps on linux - this is expected under **wayland** — compositors confine app windows to the app itself. run under an x11 session or with `qt_qpa_platform=xcb` for full always-on-top behavior. ### button disappears when switching apps on macos - fixed in 2.1.2 (the `tool` flag is now dropped on macos so the button stays visible). if it still happens, check the console log and confirm the addon is listed under **addons**. ### button is stuck minimized - fixed in 2.1.2 — the button now restores itself immediately if a system shortcut minimizes it. ### addon not in the addons menu - click **addons → refresh addons**, or restart the app. check the console for `failed to load addon ...`. ## 🪟 window / layout problems ### window too small / layout broken - the minimum window size is 800×500. maximize or resize the splitter — the sidebar is collapsible via the splitter handle. ### wrong colors / theme looks off - **view → dark mode** toggles themes. if colors look broken after a theme change, restart the app. ## 🐍 environment / venv issues ### "not running from .venv" - the app detected the system python. click **create .venv & restart** in the sidebar to bootstrap an isolated environment. ### pip install fails with externally-managed-environment (linux) - on debian/ubuntu 24.04+ the system python blocks pip. use the app's **create .venv & restart**, or create a venv manually: ```bash python3 -m venv .venv && source .venv/bin/activate && pip install -r requirements.txt ``` ## 📁 where to find logs - **windows**: `%localappdata%\ggufloader\logs` - **linux/macos**: `~/.ggufloader/logs` include the relevant log excerpt when [opening an issue](https://github.com/ggufloader/gguf-loader/issues).
    • this is the complete manual for gguf loader 2.1.2. gguf loader is a privacy-first desktop app for running large language models locally from gguf files, with zero data leaving your machine. ## 🪟 the main window the app is divided into three areas: 1. **header bar** (top) — brand on the left, a live **model status chip** on the right. 2. **sidebar** (left) — model settings, environment, and launcher sections. 3. **chat area** (center) — the conversation, or a welcome screen when no model is loaded. ## ⚙️ model settings sidebar ### model - **load gguf model** — opens a file dialog; pick any `.gguf` file. - **model info** — shows the loaded file name (or an error). ### processing - **cpu only** — runs on any machine using llama.cpp's cpu backend. - **gpu accelerated** — offloads layers to an nvidia gpu (windows/linux). requires a working cuda llama-cpp-python install. ### context length the model's context window in tokens (512–32768). larger contexts use more ram. change it before loading a model. ## 📊 environment section gguf loader ships with a built-in dependency manager for source installs: - **python · .venv status** — shows the interpreter and whether the app runs from a virtual environment. - **install missing dependencies** — appears when packages are missing; runs `pip install -r requirements.txt`. - **create .venv & restart** — bootstraps a virtual environment and relaunches inside it. - **check again** — re-scans the environment. > the packaged installers (`.exe` / linux tarball) bundle everything, so this section is mostly relevant when running from source. ## 🚀 launcher section one-click buttons that open the project's `scripts/` utilities (e.g. gpu support verification, monitor) in separate windows, plus **restart app**. ## 💬 chatting - type in the input box; **enter** sends, **shift+enter** inserts a newline. - **send** is disabled until you type something, and is disabled entirely until a model is loaded. - responses **stream** token-by-token into bubbles: your messages right (amber), ai left (charcoal). - **view → text size** (12–22) changes bubble font size live. - **file → clear chat** wipes the conversation (the model stays loaded). ## 🤖 agent mode agent mode turns the chat into a tool-using assistant that works inside a **workspace folder**: 1. toggle **🤖 agent mode: off → on**. 2. choose a workspace (combo box or 📁 browse button). default: `./agent_workspace`. 3. ask for file operations — e.g. *"create a markdown file listing today's tasks"*. the agent will: - analyze complex requests, - plan tool calls and stream status updates (🤔/💡/→/✓/✗), - execute tools against the workspace — `list_directory`, `read_file`, `write_file`, `edit_file`, `search_files`, - summarize results in natural language. **safety**: all tools are sandboxed to the workspace; paths that escape it are rejected. ## 🎨 appearance - **view → dark mode** toggles the "midnight & amber" dark theme and a light theme. - **view → text size** adjusts chat bubble text. - the app remembers dark mode per session (dark is the default). ## 🧩 addons the **addons** menu lists every loaded addon (e.g. **floating_chat**). selecting one opens it in a floating dialog; **refresh addons** re-scans the `addons/` folder. see the [addon development guide](/docs/addon-development/). ### 💬 floating chat the built-in addon adds a messenger-style floating button (always on top, draggable) that opens a chat window connected to the loaded model. it: - stays on top of all windows (see platform notes below) - remembers its position between sessions - shows model status (🟢 ready / 🔴 offline) and streams responses - has copy all / clear controls **platform notes**: fully floating on windows and linux/x11. on linux **wayland**, compositors confine it to the app window — run under x11 (`qt_qpa_platform=xcb`) for the full effect. on macos the button stays visible when the app loses focus but also appears in mission control. ## 📁 where files live - **config**: `%appdata%\ggufloader` (windows) / `~/.ggufloader` (linux/macos) - **cache**: `%localappdata%\ggufloader\cache` (windows) / `~/.cache/ggufloader` (linux) - **logs**: `%localappdata%\ggufloader\logs` (windows) / `~/.ggufloader/logs` (linux) - **addons**: the `addons/` folder next to the app ## ❤️ feedback **help → send feedback** opens the feedback dialog. point it at your own formspree endpoint via `feedback_config.json`: ```json { "endpoint_url": "https://formspree.io/f/your_form_id" } ``` ## 🆘 still stuck? see the [troubleshooting guide](/docs/troubleshooting/).

    🔗 related features

    📥

    download center get gguf loader for your platform

    🤖

    model library find the perfect model for your needs

    🎯 what's next?

    you've completed this guide! here are some suggested next steps to continue your gguf loader journey:

    🚀

    start using gguf loader

    now that you have gguf loader installed, follow our quick start guide to load your first model and start chatting.

    quick start guide →

    🤖

    browse models

    explore our curated collection of gguf models to find the perfect one for your needs.

    browse models →

    ✨

    explore features

    discover all the powerful features that make gguf loader special.

    see features →

    🌟 explore more features

    interactive demos model library community projects user stories

    🏠 back to homepage

    ✨

    explore features see what gguf loader can do

    📥

    download now get gguf loader for your system

    🤖

    browse models find the perfect model for your needs

    👥

    join community connect with other users
  • this is the complete manual for gguf loader 2.1.2. gguf loader is a privacy-first desktop app for running large language models locally from gguf files, with zero data leaving your machine. ## 🪟 the main window the app is divided into three areas: 1. **header bar** (top) — brand on the left, a live **model status chip** on the right. 2. **sidebar** (left) — model settings, environment, and launcher sections. 3. **chat area** (center) — the conversation, or a welcome screen when no model is loaded. ## ⚙️ model settings sidebar ### model - **load gguf model** — opens a file dialog; pick any `.gguf` file. - **model info** — shows the loaded file name (or an error). ### processing - **cpu only** — runs on any machine using llama.cpp's cpu backend. - **gpu accelerated** — offloads layers to an nvidia gpu (windows/linux). requires a working cuda llama-cpp-python install. ### context length the model's context window in tokens (512–32768). larger contexts use more ram. change it before loading a model. ## 📊 environment section gguf loader ships with a built-in dependency manager for source installs: - **python · .venv status** — shows the interpreter and whether the app runs from a virtual environment. - **install missing dependencies** — appears when packages are missing; runs `pip install -r requirements.txt`. - **create .venv & restart** — bootstraps a virtual environment and relaunches inside it. - **check again** — re-scans the environment. > the packaged installers (`.exe` / linux tarball) bundle everything, so this section is mostly relevant when running from source. ## 🚀 launcher section one-click buttons that open the project's `scripts/` utilities (e.g. gpu support verification, monitor) in separate windows, plus **restart app**. ## 💬 chatting - type in the input box; **enter** sends, **shift+enter** inserts a newline. - **send** is disabled until you type something, and is disabled entirely until a model is loaded. - responses **stream** token-by-token into bubbles: your messages right (amber), ai left (charcoal). - **view → text size** (12–22) changes bubble font size live. - **file → clear chat** wipes the conversation (the model stays loaded). ## 🤖 agent mode agent mode turns the chat into a tool-using assistant that works inside a **workspace folder**: 1. toggle **🤖 agent mode: off → on**. 2. choose a workspace (combo box or 📁 browse button). default: `./agent_workspace`. 3. ask for file operations — e.g. *"create a markdown file listing today's tasks"*. the agent will: - analyze complex requests, - plan tool calls and stream status updates (🤔/💡/→/✓/✗), - execute tools against the workspace — `list_directory`, `read_file`, `write_file`, `edit_file`, `search_files`, - summarize results in natural language. **safety**: all tools are sandboxed to the workspace; paths that escape it are rejected. ## 🎨 appearance - **view → dark mode** toggles the "midnight & amber" dark theme and a light theme. - **view → text size** adjusts chat bubble text. - the app remembers dark mode per session (dark is the default). ## 🧩 addons the **addons** menu lists every loaded addon (e.g. **floating_chat**). selecting one opens it in a floating dialog; **refresh addons** re-scans the `addons/` folder. see the [addon development guide](/docs/addon-development/). ### 💬 floating chat the built-in addon adds a messenger-style floating button (always on top, draggable) that opens a chat window connected to the loaded model. it: - stays on top of all windows (see platform notes below) - remembers its position between sessions - shows model status (🟢 ready / 🔴 offline) and streams responses - has copy all / clear controls **platform notes**: fully floating on windows and linux/x11. on linux **wayland**, compositors confine it to the app window — run under x11 (`qt_qpa_platform=xcb`) for the full effect. on macos the button stays visible when the app loses focus but also appears in mission control. ## 📁 where files live - **config**: `%appdata%\ggufloader` (windows) / `~/.ggufloader` (linux/macos) - **cache**: `%localappdata%\ggufloader\cache` (windows) / `~/.cache/ggufloader` (linux) - **logs**: `%localappdata%\ggufloader\logs` (windows) / `~/.ggufloader/logs` (linux) - **addons**: the `addons/` folder next to the app ## ❤️ feedback **help → send feedback** opens the feedback dialog. point it at your own formspree endpoint via `feedback_config.json`: ```json { "endpoint_url": "https://formspree.io/f/your_form_id" } ``` ## 🆘 still stuck? see the [troubleshooting guide](/docs/troubleshooting/).
  • addon development guide

    learn to create custom addons for gguf loader with examples and best practices

    advanced 15 minutes

    gguf loader’s addon system lets you extend the app without touching its source. addons are python packages dropped into the addons/ folder; the app discovers, loads, and manages them automatically.

    📦 what an addon looks like

    addons/
    └── my_addon/
        ├── __init__.py      # must expose register()
        ├── main.py          # your logic (any structure you like)
        └── ...              # widgets, resources, etc.
    

    minimum viable addon

    # addons/my_addon/__init__.py
    from pyside6.qtwidgets import qlabel
    
    def register(parent=none):
        """called by gguf loader when the addon loads."""
        return qlabel("hello from my addon!")
    

    that’s it — drop the folder in addons/, restart (or addons → refresh addons), and your widget appears in the addons menu.

    🔍 how loading works

    addonmanager (addon_manager.py):

    1. scan_addons() — lists subdirectories of the addons folder that contain __init__.py.
    2. load_addon(name, path) — dynamically imports the module (importlib.util.spec_from_file_location) and requires a register attribute.
    3. load_all_addons() — loads every addon and records success/failure in the console log.
    4. get_addon_widget(name, parent) — calls register(parent) and returns the widget.
    5. open_addon_dialog(name, parent) — shows the addon widget in a non-modal dialog from the addons menu.

    the main window calls get_addon_widget for each loaded addon at startup, so register() runs even before the user opens the menu — this is how background addons (like the floating chat button) start automatically.

    📡 talking to the app

    your register(parent) receives the parent widget. the main window exposes an addon-facing api (see the addon api reference). the robust pattern used by the built-in addon:

    def register(parent=none):
        gguf_app = none
        # 1. parent might already be the main window
        if parent and hasattr(parent, 'model') and hasattr(parent, 'model_loaded'):
            gguf_app = parent
        else:
            # 2. walk up the parent chain
            current = parent
            while current is not none:
                if hasattr(current, 'model') and hasattr(current, 'model_loaded'):
                    gguf_app = current
                    break
                current = current.parent()
            # 3. fall back to scanning top-level widgets
            if gguf_app is none:
                app = qapplication.instance()
                for widget in app.toplevelwidgets():
                    if hasattr(widget, 'model') and hasattr(widget, 'model_loaded'):
                        gguf_app = widget
                        break
        ...
    

    key api members: model (callable backend), model_loaded / model_unloaded / generation_finished / generation_error signals, addon_manager, and _floating_chat_addon.

    🤖 calling the model

    # streaming
    for token in gguf_app.model(prompt, stream=true, max_tokens=512):
        print(token["choices"][0]["text"])
    
    # or use the backend directly
    response = gguf_app.model.generate(prompt, max_tokens=512, temperature=0.7)
    

    model is none until the user loads a model — always check, and subscribe to model_loaded to react.

    🧩 widget vs. background addons

    • return a widget from register() → appears in the addons menu, opens in a dialog.
    • return none (or also keep your own top-level windows) → runs in the background. the floating chat addon returns a small status widget but drives its own always-on-top windows.

    💡 best practices

    1. guard everything — wrap registration in try/except and log failures; a broken addon must never crash the app.
    2. clean up state — if you keep references on the main window (e.g. gguf_app._my_addon = ...), stop them on app close (mainwindow.closeevent stops _floating_chat_addon — follow that pattern).
    3. use qsettings for persistence — qsettings("ggufloader", "youraddon").
    4. prefer signals — let your widgets emit signals; don’t reach into other widgets’ internals.
    5. qt-free logic — if you have non-ui logic (parsing, computation), put it in a module with no qt imports so it’s unit-testable.

    ✅ debugging

    addon load errors print to the console with a traceback (failed to load addon <name>: ...). run the app from a terminal to see them:

    python main.py
    

    📚 next steps

    • addon api reference — every hook and member
    • floating chat example — full walkthrough of the built-in addon

    related documentation

    • addon api reference

      complete api reference for developing gguf loader addons

      advanced 20 minutes

      complete reference for the hooks, signals, and objects available to gguf loader addons.

      🪝 addon contract

      an addon is a folder under addons/ with an __init__.py exposing:

      register(parent=none) -> qwidget | none

      called by addonmanager when the addon loads. returns an optional widget shown in the addons menu (opened in a non-modal dialog), or none for background-only addons.

      parameters — parent: the widget the addon is being attached to (may be the main window, a dialog, or another widget).

      return — a widget, or none.

      def register(parent=none):
          return qlabel("my addon")   # widget addon
      

      🖥️ main window api

      addons receive a reference to the main window (via parent or by scanning qapplication.instance().toplevelwidgets()). it exposes:

      properties

      | member | type | description | |—|—|—| | model | modelbackend \| none | the loaded model backend; callable. none until a model is loaded. | | chat_generator | none | legacy hook; always none. addons should call model instead. | | addon_manager | addonmanager | the addon manager instance. | | _floating_chat_addon | floatingchataddon \| none | the built-in floating chat addon (if running). |

      signals

      | signal | signature | description | |—|—|—| | model_loaded | signal(object) | emitted with the modelbackend after a model loads. | | model_unloaded | signal() | emitted after the model is released. | | generation_finished | signal() | emitted when a chat generation completes. | | generation_error | signal(str) | emitted with an error message when generation fails. | | theme_changed | signal(bool) | emitted when dark mode toggles (true = dark). |

      gguf_app.model_loaded.connect(self._on_model_loaded)
      gguf_app.generation_finished.connect(self._on_finished)
      

      🧠 modelbackend

      gguf_app.model is a core.llm.model_backend.modelbackend. it is callable and thread-safe (all access serialized through a lock).

      calling convention

      # raw llama-cpp shape (addon-compatible)
      result = gguf_app.model(prompt, stream=false, max_tokens=512, temperature=0.7)
      text = result["choices"][0]["text"]
      
      # streaming
      for token_data in gguf_app.model(prompt, stream=true, max_tokens=512):
          text = token_data["choices"][0]["text"]
      

      methods & attributes

      | member | description | |—|—| | model_path | path of the loaded gguf file. | | use_gpu | whether gpu acceleration is active. | | n_ctx | context length. | | is_loaded | true once the runtime exists. | | load() | create the underlying llama runtime (raises on failure). | | unload() | release the model and free memory. | | generate(prompt, **kwargs) | complete non-streamed response string. | | generate_stream(prompt, **kwargs) | iterator of token strings. | | __call__(prompt, **kwargs) | llama-cpp-compatible callable (stream returns token-dict generator). |

      kwargs are passed straight to llama_cpp: max_tokens, temperature, top_p, top_k, repeat_penalty, stop, etc.

      🧰 agent tools (for agent-capable addons)

      the agent engine (core/agent/) exposes sandboxed tools through toolregistry. tools are not qt-bound and are safe to call from any thread.

      tool	params	result
      list_directory	path (default .)	{status, result: [{name, type, size}]}
      read_file	path, max_size, encoding	{status, result: content, lines, encoding}
      write_file	path, content	{status, path, bytes_written}
      edit_file	path, operation (replace/insert_line/delete_line), find, replace, line_number, content	{status, operation, changes_made}
      search_files	pattern/query, path	{status, result: [paths], total_matches}

      every tool result includes status ("success" / "error") and tool_name. all paths are sandboxed to the workspace — escaping paths raise.

      from core.agent.tool_registry import create_default_registry
      registry = create_default_registry("./workspace")
      result = registry.execute("list_directory", {"path": "."})
      

      🧩 addonmanager

      method	description
      scan_addons()	{name: path} of addons with __init__.py.
      load_addon(name, path)	import + validate register; true/false.
      load_all_addons()	load all; {name: success}.
      get_addon_widget(name, parent)	widget from register(parent).
      open_addon_dialog(name, parent)	show addon in a non-modal dialog.
      get_loaded_addons()	list of successfully loaded names.

      💾 persistence

      use qt’s qsettings with a scoped org/app key to avoid collisions:

      from pyside6.qtcore import qsettings
      settings = qsettings("ggufloader", "myaddon")
      settings.setvalue("position", point)
      pos = settings.value("position", qpoint(100, 100))
      

      ⚡ lifecycle

      • register() is called at startup and on addons → refresh addons.
      • the main window’s closeevent stops the floating chat addon; follow that pattern to stop your own background objects:

        if hasattr(gguf_app, '_my_addon') and gguf_app._my_addon:
            gguf_app._my_addon.stop()
        

      📚 see also

      • addon development guide — tutorial
      • floating chat example — full real-world addon

      related documentation

      • gguf loader's addon system lets you extend the app without touching its source. addons are python packages dropped into the `addons/` folder; the app discovers, loads, and manages them automatically. ## 📦 what an addon looks like ``` addons/ └── my_addon/ ├── __init__.py # must expose register() ├── main.py # your logic (any structure you like) └── ... # widgets, resources, etc. ``` ### minimum viable addon ```python # addons/my_addon/__init__.py from pyside6.qtwidgets import qlabel def register(parent=none): """called by gguf loader when the addon loads.""" return qlabel("hello from my addon!") ``` that's it — drop the folder in `addons/`, restart (or **addons → refresh addons**), and your widget appears in the **addons** menu. ## 🔍 how loading works `addonmanager` (`addon_manager.py`): 1. `scan_addons()` — lists subdirectories of the addons folder that contain `__init__.py`. 2. `load_addon(name, path)` — dynamically imports the module (`importlib.util.spec_from_file_location`) and requires a `register` attribute. 3. `load_all_addons()` — loads every addon and records success/failure in the console log. 4. `get_addon_widget(name, parent)` — calls `register(parent)` and returns the widget. 5. `open_addon_dialog(name, parent)` — shows the addon widget in a non-modal dialog from the addons menu. the main window calls `get_addon_widget` for each loaded addon at startup, so `register()` runs even before the user opens the menu — this is how background addons (like the floating chat button) start automatically. ## 📡 talking to the app your `register(parent)` receives the parent widget. the **main window** exposes an addon-facing api (see the [addon api reference](/docs/addon-api/)). the robust pattern used by the built-in addon: ```python def register(parent=none): gguf_app = none # 1. parent might already be the main window if parent and hasattr(parent, 'model') and hasattr(parent, 'model_loaded'): gguf_app = parent else: # 2. walk up the parent chain current = parent while current is not none: if hasattr(current, 'model') and hasattr(current, 'model_loaded'): gguf_app = current break current = current.parent() # 3. fall back to scanning top-level widgets if gguf_app is none: app = qapplication.instance() for widget in app.toplevelwidgets(): if hasattr(widget, 'model') and hasattr(widget, 'model_loaded'): gguf_app = widget break ... ``` key api members: `model` (callable backend), `model_loaded` / `model_unloaded` / `generation_finished` / `generation_error` signals, `addon_manager`, and `_floating_chat_addon`. ## 🤖 calling the model ```python # streaming for token in gguf_app.model(prompt, stream=true, max_tokens=512): print(token["choices"][0]["text"]) # or use the backend directly response = gguf_app.model.generate(prompt, max_tokens=512, temperature=0.7) ``` `model` is `none` until the user loads a model — always check, and subscribe to `model_loaded` to react. ## 🧩 widget vs. background addons - **return a widget** from `register()` → appears in the addons menu, opens in a dialog. - **return `none`** (or also keep your own top-level windows) → runs in the background. the floating chat addon returns a small status widget but drives its own always-on-top windows. ## 💡 best practices 1. **guard everything** — wrap registration in `try/except` and log failures; a broken addon must never crash the app. 2. **clean up state** — if you keep references on the main window (e.g. `gguf_app._my_addon = ...`), stop them on app close (`mainwindow.closeevent` stops `_floating_chat_addon` — follow that pattern). 3. **use qsettings for persistence** — `qsettings("ggufloader", "youraddon")`. 4. **prefer signals** — let your widgets emit signals; don't reach into other widgets' internals. 5. **qt-free logic** — if you have non-ui logic (parsing, computation), put it in a module with no qt imports so it's unit-testable. ## ✅ debugging addon load errors print to the console with a traceback (`failed to load addon : ...`). run the app from a terminal to see them: ```bash python main.py ``` ## 📚 next steps - [addon api reference](/docs/addon-api/) — every hook and member - [floating chat example](/docs/floating-chat-example/) — full walkthrough of the built-in addon
      • this page walks through **floating_chat**, gguf loader's built-in addon — a facebook messenger-style floating button that opens a chat window connected to the loaded model. it's the best real-world reference for writing a background addon with its own windows. ``` addons/floating_chat/ ├── __init__.py # exposes register() ├── main.py # floatingchataddon (qobject) + register() ├── floating_button.py # floatingchatbutton - draggable always-on-top button ├── chat_window.py # floatingchatwindow - the chat ui └── status_widget.py # small sidebar widget returned by register() ``` ## 🧠 the design the addon is a `qobject` (`floatingchataddon`) that owns two windows: - **`floatingchatbutton`** — a frameless, always-on-top tool window that can be dragged anywhere and remembers its position via `qsettings`. - **`floatingchatwindow`** — a regular top-level window (`400×600`) positioned next to the button, with a message list, streaming replies, and copy all / clear buttons. ## 🪝 registration `__init__.py` simply re-exports `register`: ```python from .main import register __all__ = ['register'] ``` `register()` finds the main window (parent → parent chain → `toplevelwidgets()` scan), stops any previous instance, starts the addon, and returns a **status widget** for the sidebar: ```python def register(parent=none): gguf_app = _find_main_window(parent) # model + model_loaded check if gguf_app is none: return none if getattr(gguf_app, '_floating_chat_addon', none): gguf_app._floating_chat_addon.stop() addon = floatingchataddon(gguf_app) if addon.start(): gguf_app._floating_chat_addon = addon return floatingchatstatuswidget(addon) return none ``` the returned `floatingchatstatuswidget` shows up in the **addons** menu, and because `register()` runs at startup, the button appears automatically — no user action needed. ## 📡 connecting to the app `floatingchataddon.__init__` subscribes to model lifecycle signals: ```python if hasattr(self.gguf_app, 'model_loaded'): self.gguf_app.model_loaded.connect(self._on_model_loaded) if hasattr(self.gguf_app, 'generation_finished'): self.gguf_app.generation_finished.connect(self._on_generation_finished) if hasattr(self.gguf_app, 'generation_error'): self.gguf_app.generation_error.connect(self._on_generation_error) ``` `_on_model_loaded` flips the chat window's status to **🟢 model: ready**; generation events update the streaming bubble / error state. ## 🏃 lifecycle: start() / stop() ```python def start(self): self._floating_button = floatingchatbutton() self._floating_button.clicked.connect(self._on_button_clicked) self._load_button_position() # qsettings restore, clamped to screen self._floating_button.show() self.addon_started.emit() return true def stop(self): self._save_button_position() # persist for next session if self._chat_window: self._chat_window.close() if self._floating_button: self._floating_button.close() self.addon_stopped.emit() ``` `stop()` is idempotent and called on app close via `mainwindow.closeevent`, so there's never a dangling button. ## 🖱️ button click → window toggle the click handler distinguishes three states (a minimized window is "visible" to qt but off-screen, so it must be restored, not hidden): ```python def _on_button_clicked(self): chat = self._chat_window if chat and chat.isvisible(): if chat.isminimized(): chat.shownormal(); chat.raise_(); chat.activatewindow() else: chat.hide() else: self._show_chat_window() ``` `_show_chat_window` creates the window lazily, positions it next to the button (flipping to the left / up when it would overflow the screen's **available geometry**, which excludes taskbars and docks), then `shownormal()` + `raise_()` + `activatewindow()`. ## 🪟 window flags (the platform subtleties) ```python flags = (qt.windowtype.framelesswindowhint | qt.windowtype.windowstaysontophint | qt.windowtype.x11bypasswindowmanagerhint) # linux/x11 if sys.platform != "darwin": flags |= qt.windowtype.tool # keeps it out of the taskbar ``` - **windows/linux**: `tool` keeps the button out of the taskbar. `x11bypasswindowmanagerhint` gives true always-on-top under x11. - **macos**: the `tool` flag would turn the button into a utility window that **auto-hides when the app loses focus** — so it's deliberately dropped. trade-off: the button appears in mission control. - **wayland**: compositors don't allow floating above *other* apps' windows; the button is confined to the app window (documented, and x11 is recommended for the full experience). the chat window drops the minimize button entirely — a minimized companion window is a trap state that can't be reliably restored. ## 📌 resiliancy touches - `changeevent` bounces the button straight back to normal if a system shortcut (e.g. win+d) minimizes *every* window including the tool: ```python if event.type() == qevent.type.windowstatechange and self.isminimized(): qtimer.singleshot(0, self.shownormal) ``` - dragging clamps to `screen.availablegeometry()` (not `(0,0)`), so the button can't slide under the macos menu bar or a windows taskbar. - saved positions are clamped the same way on restore. ## 🧩 what you can reuse - the **window-finding pattern** in `register()` — works for any addon. - the **qsettings position persistence** with on-screen clamping. - the **platform-conditional window flags** — copy it for any always-on-top addon window. - the **show/hide/restore toggle** logic. ## 📚 see also - [addon development guide](/docs/addon-development/) - [addon api reference](/docs/addon-api/)
      • this page documents how the gguf loader codebase is organized. the app is a pyside6 desktop application; a single `main.py` bootstraps everything, and logic is split into `core/` (pure logic), `services/` (qt threading bridges), and `ui/` + `widgets/` (presentation). ## 🗂️ top-level layout ``` gguf-loader/ ├── main.py # entry point: logging, dll paths, qapplication, mainwindow ├── config.py # central configuration constants ├── resource_manager.py # resource/path discovery (dev, package, or frozen) ├── addon_manager.py # loads and manages addons ├── requirements.txt # python dependencies ├── build_exe.spec # pyinstaller spec used for windows & linux builds │ ├── core/ # pure, testable logic (no qt) │ ├── llm/ │ │ ├── model_backend.py # thread-safe llama-cpp-python wrapper │ │ └── prompt_builder.py # system-prompt & conversation assembly │ └── agent/ │ ├── agent_engine.py # tool-use agent loop (no qt, no llama_cpp) │ └── tool_registry.py # sandboxed workspace tools │ ├── services/ # qobject bridges that run work on threads │ ├── model_service.py # load/unload models on a qthread │ ├── chat_service.py # streaming generation on a qthread │ ├── agent_service.py # runs agentengine on a worker thread │ ├── environment_service.py # venv/dependency checks & pip tasks │ └── launcher_service.py # launches scripts/ utilities │ ├── ui/ # main window & panels │ ├── main_window.py # composition root + addon-facing api │ ├── chat_panel.py # chat display, input, agent controls │ ├── sidebar_panel.py # model settings sidebar │ └── theme.py # dark/light qss token system │ ├── widgets/ # reusable widgets │ ├── chat_bubble.py # chatgpt-style bubble │ └── feedback_dialog.py # feedback form dialog │ ├── addons/ # addon packages (each has __init__.py with register()) │ └── floating_chat/ # built-in floating chat addon │ ├── scripts/ # utility & release scripts │ ├── capture_screenshots.py # regenerates readme/site screenshots │ ├── install_linux.sh # linux installer/uninstaller │ ├── package_linux.sh # builds the linux .tar.gz release │ └── ... # gpu install/monitor helpers │ └── build_hooks/ # pyinstaller hook modules ``` ## 🔁 key design rules - **`core/` never imports qt or llama_cpp.** it receives plain callables, so it can be unit-tested in isolation. - **`services/` are the only place qt threads are created.** ui never spins up threads directly. - **`ui/` widgets are "dumb"** — they render state and emit signals; `mainwindow` owns all logic. - **`resource_manager.py`** makes paths work identically in dev, as an installed package, and in a pyinstaller bundle (`sys._meipass`). ## 🧵 threading model ``` ui thread (mainwindow) worker thread │ │ │── modelservice.load() ─────→│ qthread: llama_cpp loads model │←────── loaded(modelbackend) │ │── chatservice.generate() ──→│ qthread: streams tokens │←────── token_received(text) │ │── agentservice.process() ──→│ qthread: agentengine tool loop │←────── status/tool/response │ ``` a fresh `qthread` + worker is created per request (the professional qt pattern — `qthread` is never subclassed). ## 🚀 entry point flow `main.py` → `setup_library_path()` (finds bundled llama.cpp libs) → `qapplication` → `mainwindow()` → builds ui, wires services, checks environment, loads addons → `app.exec()`. ## 📦 packaging - **windows**: `build_exe.bat` / `build_exe.spec` → `ggufloader_vx.y.z.exe` - **linux**: build the spec inside a linux environment, then `scripts/package_linux.sh` wraps the binary + installer + icon into a `.tar.gz` - github actions publishes both to every release automatically. see the [architecture overview](/docs/architecture/) for deeper design rationale.

      🔗 related features

      👥

      community hub connect with other developers

      🛠️

      addon showcase see community creations

      🎯 what's next?

      you've completed this guide! here are some suggested next steps to continue your gguf loader journey:

      🚀

      share your addon

      built something awesome? share it with the community and get featured on our homepage.

      share with community →

      📖

      study examples

      learn from real-world addon examples and see what's possible.

      addon showcase →

      🤝

      get support

      need help with your addon? our community is here to support you.

      get help →

      🌟 explore more features

      interactive demos model library community projects user stories

      🏠 back to homepage

      ✨

      explore features see what gguf loader can do

      📥

      download now get gguf loader for your system

      🤖

      browse models find the perfect model for your needs

      👥

      join community connect with other users
    • this page walks through **floating_chat**, gguf loader's built-in addon — a facebook messenger-style floating button that opens a chat window connected to the loaded model. it's the best real-world reference for writing a background addon with its own windows. ``` addons/floating_chat/ ├── __init__.py # exposes register() ├── main.py # floatingchataddon (qobject) + register() ├── floating_button.py # floatingchatbutton - draggable always-on-top button ├── chat_window.py # floatingchatwindow - the chat ui └── status_widget.py # small sidebar widget returned by register() ``` ## 🧠 the design the addon is a `qobject` (`floatingchataddon`) that owns two windows: - **`floatingchatbutton`** — a frameless, always-on-top tool window that can be dragged anywhere and remembers its position via `qsettings`. - **`floatingchatwindow`** — a regular top-level window (`400×600`) positioned next to the button, with a message list, streaming replies, and copy all / clear buttons. ## 🪝 registration `__init__.py` simply re-exports `register`: ```python from .main import register __all__ = ['register'] ``` `register()` finds the main window (parent → parent chain → `toplevelwidgets()` scan), stops any previous instance, starts the addon, and returns a **status widget** for the sidebar: ```python def register(parent=none): gguf_app = _find_main_window(parent) # model + model_loaded check if gguf_app is none: return none if getattr(gguf_app, '_floating_chat_addon', none): gguf_app._floating_chat_addon.stop() addon = floatingchataddon(gguf_app) if addon.start(): gguf_app._floating_chat_addon = addon return floatingchatstatuswidget(addon) return none ``` the returned `floatingchatstatuswidget` shows up in the **addons** menu, and because `register()` runs at startup, the button appears automatically — no user action needed. ## 📡 connecting to the app `floatingchataddon.__init__` subscribes to model lifecycle signals: ```python if hasattr(self.gguf_app, 'model_loaded'): self.gguf_app.model_loaded.connect(self._on_model_loaded) if hasattr(self.gguf_app, 'generation_finished'): self.gguf_app.generation_finished.connect(self._on_generation_finished) if hasattr(self.gguf_app, 'generation_error'): self.gguf_app.generation_error.connect(self._on_generation_error) ``` `_on_model_loaded` flips the chat window's status to **🟢 model: ready**; generation events update the streaming bubble / error state. ## 🏃 lifecycle: start() / stop() ```python def start(self): self._floating_button = floatingchatbutton() self._floating_button.clicked.connect(self._on_button_clicked) self._load_button_position() # qsettings restore, clamped to screen self._floating_button.show() self.addon_started.emit() return true def stop(self): self._save_button_position() # persist for next session if self._chat_window: self._chat_window.close() if self._floating_button: self._floating_button.close() self.addon_stopped.emit() ``` `stop()` is idempotent and called on app close via `mainwindow.closeevent`, so there's never a dangling button. ## 🖱️ button click → window toggle the click handler distinguishes three states (a minimized window is "visible" to qt but off-screen, so it must be restored, not hidden): ```python def _on_button_clicked(self): chat = self._chat_window if chat and chat.isvisible(): if chat.isminimized(): chat.shownormal(); chat.raise_(); chat.activatewindow() else: chat.hide() else: self._show_chat_window() ``` `_show_chat_window` creates the window lazily, positions it next to the button (flipping to the left / up when it would overflow the screen's **available geometry**, which excludes taskbars and docks), then `shownormal()` + `raise_()` + `activatewindow()`. ## 🪟 window flags (the platform subtleties) ```python flags = (qt.windowtype.framelesswindowhint | qt.windowtype.windowstaysontophint | qt.windowtype.x11bypasswindowmanagerhint) # linux/x11 if sys.platform != "darwin": flags |= qt.windowtype.tool # keeps it out of the taskbar ``` - **windows/linux**: `tool` keeps the button out of the taskbar. `x11bypasswindowmanagerhint` gives true always-on-top under x11. - **macos**: the `tool` flag would turn the button into a utility window that **auto-hides when the app loses focus** — so it's deliberately dropped. trade-off: the button appears in mission control. - **wayland**: compositors don't allow floating above *other* apps' windows; the button is confined to the app window (documented, and x11 is recommended for the full experience). the chat window drops the minimize button entirely — a minimized companion window is a trap state that can't be reliably restored. ## 📌 resiliancy touches - `changeevent` bounces the button straight back to normal if a system shortcut (e.g. win+d) minimizes *every* window including the tool: ```python if event.type() == qevent.type.windowstatechange and self.isminimized(): qtimer.singleshot(0, self.shownormal) ``` - dragging clamps to `screen.availablegeometry()` (not `(0,0)`), so the button can't slide under the macos menu bar or a windows taskbar. - saved positions are clamped the same way on restore. ## 🧩 what you can reuse - the **window-finding pattern** in `register()` — works for any addon. - the **qsettings position persistence** with on-screen clamping. - the **platform-conditional window flags** — copy it for any always-on-top addon window. - the **show/hide/restore toggle** logic. ## 📚 see also - [addon development guide](/docs/addon-development/) - [addon api reference](/docs/addon-api/)
    • this page documents how the gguf loader codebase is organized. the app is a pyside6 desktop application; a single `main.py` bootstraps everything, and logic is split into `core/` (pure logic), `services/` (qt threading bridges), and `ui/` + `widgets/` (presentation). ## 🗂️ top-level layout ``` gguf-loader/ ├── main.py # entry point: logging, dll paths, qapplication, mainwindow ├── config.py # central configuration constants ├── resource_manager.py # resource/path discovery (dev, package, or frozen) ├── addon_manager.py # loads and manages addons ├── requirements.txt # python dependencies ├── build_exe.spec # pyinstaller spec used for windows & linux builds │ ├── core/ # pure, testable logic (no qt) │ ├── llm/ │ │ ├── model_backend.py # thread-safe llama-cpp-python wrapper │ │ └── prompt_builder.py # system-prompt & conversation assembly │ └── agent/ │ ├── agent_engine.py # tool-use agent loop (no qt, no llama_cpp) │ └── tool_registry.py # sandboxed workspace tools │ ├── services/ # qobject bridges that run work on threads │ ├── model_service.py # load/unload models on a qthread │ ├── chat_service.py # streaming generation on a qthread │ ├── agent_service.py # runs agentengine on a worker thread │ ├── environment_service.py # venv/dependency checks & pip tasks │ └── launcher_service.py # launches scripts/ utilities │ ├── ui/ # main window & panels │ ├── main_window.py # composition root + addon-facing api │ ├── chat_panel.py # chat display, input, agent controls │ ├── sidebar_panel.py # model settings sidebar │ └── theme.py # dark/light qss token system │ ├── widgets/ # reusable widgets │ ├── chat_bubble.py # chatgpt-style bubble │ └── feedback_dialog.py # feedback form dialog │ ├── addons/ # addon packages (each has __init__.py with register()) │ └── floating_chat/ # built-in floating chat addon │ ├── scripts/ # utility & release scripts │ ├── capture_screenshots.py # regenerates readme/site screenshots │ ├── install_linux.sh # linux installer/uninstaller │ ├── package_linux.sh # builds the linux .tar.gz release │ └── ... # gpu install/monitor helpers │ └── build_hooks/ # pyinstaller hook modules ``` ## 🔁 key design rules - **`core/` never imports qt or llama_cpp.** it receives plain callables, so it can be unit-tested in isolation. - **`services/` are the only place qt threads are created.** ui never spins up threads directly. - **`ui/` widgets are "dumb"** — they render state and emit signals; `mainwindow` owns all logic. - **`resource_manager.py`** makes paths work identically in dev, as an installed package, and in a pyinstaller bundle (`sys._meipass`). ## 🧵 threading model ``` ui thread (mainwindow) worker thread │ │ │── modelservice.load() ─────→│ qthread: llama_cpp loads model │←────── loaded(modelbackend) │ │── chatservice.generate() ──→│ qthread: streams tokens │←────── token_received(text) │ │── agentservice.process() ──→│ qthread: agentengine tool loop │←────── status/tool/response │ ``` a fresh `qthread` + worker is created per request (the professional qt pattern — `qthread` is never subclassed). ## 🚀 entry point flow `main.py` → `setup_library_path()` (finds bundled llama.cpp libs) → `qapplication` → `mainwindow()` → builds ui, wires services, checks environment, loads addons → `app.exec()`. ## 📦 packaging - **windows**: `build_exe.bat` / `build_exe.spec` → `ggufloader_vx.y.z.exe` - **linux**: build the spec inside a linux environment, then `scripts/package_linux.sh` wraps the binary + installer + icon into a `.tar.gz` - github actions publishes both to every release automatically. see the [architecture overview](/docs/architecture/) for deeper design rationale.

    🔗 related features

    👥

    community hub connect with other developers

    🛠️

    addon showcase see community creations

    🎯 what's next?

    you've completed this guide! here are some suggested next steps to continue your gguf loader journey:

    🚀

    share your addon

    built something awesome? share it with the community and get featured on our homepage.

    share with community →

    📖

    study examples

    learn from real-world addon examples and see what's possible.

    addon showcase →

    🤝

    get support

    need help with your addon? our community is here to support you.

    get help →

    🌟 explore more features

    interactive demos model library community projects user stories

    🏠 back to homepage

    ✨

    explore features see what gguf loader can do

    📥

    download now get gguf loader for your system

    🤖

    browse models find the perfect model for your needs

    👥

    join community connect with other users

  🔗 related features

  ✨

  feature showcase see what gguf loader can do

  📚

  how-to guides learn with interactive tutorials

  🎯 what's next?

  you've completed this guide! here are some suggested next steps to continue your gguf loader journey:

  🛠️

  create your first addon

  ready to extend gguf loader? learn how to create custom addons with our development guide.

  addon development →

  👥

  join the community

  connect with other gguf loader users, share your projects, and get help.

  community hub →

  📚

  advanced tutorials

  take your skills to the next level with advanced how-to guides and tutorials.

  how-to guides →

  🌟 explore more features

  interactive demos model library community projects user stories

  🏠 back to homepage

  ✨

  explore features see what gguf loader can do

  📥

  download now get gguf loader for your system

  🤖

  browse models find the perfect model for your needs

  👥

  join community connect with other users
• installation guide

  complete guide to installing gguf loader on windows, macos, and linux

  beginner 5 minutes

  this guide will help you install gguf loader 2.2.0 on your system. want to see what gguf loader can do first? explore the features on our homepage.

  📋 system requirements

  • os: windows 10/11, linux (x86_64), or macos
  • ram: 4 gb minimum (8 gb+ recommended for larger models)
  • storage: 2 gb free space for the app, plus room for model files
  • gpu: optional — cpu-only works everywhere; nvidia cuda is supported on windows and linux
  • python: not required for the installers (everything is bundled)

  🚀 quick installation

  the prebuilt installers bundle python, pyside6, and llama.cpp — you do not need python installed.

  windows

  1. download ggufloader_v2.2.0_cpu.exe (~69 mb) from the releases page. this is the cpu-only build that works on any pc.
  2. have an nvidia gpu with cuda? grab ggufloader_v2.2.0_gpu.exe (~854 mb) instead for gpu acceleration.
  3. double-click the downloaded file and run it. no installation wizard — the app starts immediately.

  ⚠️ windows smartscreen: the first launch may show “windows protected your pc”. click more info → run anyway. this is normal for unsigned open-source binaries.

  linux (x86_64)

  1. download ggufloader_v2.2.0_linux_x86_64_cpu (~121 mb) from the releases page.
  2. make it executable and run:

     chmod +x ggufloader_v2.2.0_linux_x86_64_cpu
     ./ggufloader_v2.2.0_linux_x86_64_cpu
     

     no installation needed — the app starts directly. everything (python, pyside6, llama.cpp) is bundled.

  macos

  no prebuilt installer is published yet. run from source (below) — everything works on macos.

  run from source (any platform)

  for development, or to run on an unsupported platform:

  git clone https://github.com/ggufloader/gguf-loader.git
  cd gguf-loader
  pip install -r requirements.txt
  python main.py
  

  ✅ verify your installation

  run ggufloader_v2.2.0_cpu.exe --version (windows) or ./ggufloader_v2.2.0_linux_x86_64_cpu --version (linux) — it prints gguf loader version 2.2.0 and exits.

  🔧 next steps

  • quick start guide — load your first model
  • troubleshooting — if something isn’t working

  related documentation

  • get up and running with gguf loader 2.2.0 in just a few minutes! new to gguf loader? [check out the homepage](/) to see what makes it special. ## 🚀 step 1: install gguf loader follow the [installation guide](/docs/installation/) for your platform, or download directly: - **windows**: [ggufloader_v2.2.0_cpu.exe](https://github.com/ggufloader/gguf-loader/releases/download/v2.2.0/ggufloader_v2.2.0_cpu.exe) (cpu-only) · [ggufloader_v2.2.0_gpu.exe](https://github.com/ggufloader/gguf-loader/releases/download/v2.2.0/ggufloader_v2.2.0_gpu.exe) (nvidia cuda) - **linux**: [ggufloader_v2.2.0_linux_x86_64_cpu](https://github.com/ggufloader/gguf-loader/releases/download/v2.2.0/ggufloader_v2.2.0_linux_x86_64_cpu) ## 📥 step 2: get a gguf model gguf loader runs any gguf-format model. good starter models on hugging face: - **small (fast)**: [llama-3-8b-instruct gguf (q4_0)](https://huggingface.co/thebloke/llama-3-8b-instruct-gguf) — ~4.7 gb, runs great on cpu - **medium**: [mistral-7b-instruct gguf (q4_k_m)](https://huggingface.co/thebloke/mistral-7b-instruct-v0.2-gguf) — ~4.1 gb - **larger**: [phi-3.5-mini-instruct gguf](https://huggingface.co/microsoft/phi-3.5-mini-instruct-gguf) or any qwen/llama gguf you like download the `.gguf` file and remember where you saved it. ## 🖥️ step 3: load the model 1. launch gguf loader. 2. in the **model settings** sidebar, choose **processing** mode — `cpu only` (default, works everywhere) or `gpu accelerated` (nvidia cuda). 3. set a **context length** (8192 is a safe default for most models; 32768 uses more ram). 4. click **load gguf model** and select your `.gguf` file. 5. wait for the status to show **"model ready!"** — the header chip turns green with the model name. ## 💬 step 4: chat type a message in the input box at the bottom and press **enter** (shift+enter inserts a newline). responses stream in as chatgpt-style bubbles — your messages on the right (amber), the ai's on the left. use the **view → text size** menu to adjust bubble font size. ## 🤖 step 5 (optional): try agent mode click **🤖 agent mode: off** in the input area to toggle it on: 1. pick a **workspace folder** (defaults to `./agent_workspace`). 2. ask the agent to do file work — e.g. *"create a file called hello.py that prints 'hi'"*. 3. the agent plans tool calls, executes them (read/write/edit/search files inside the workspace only), and reports back with live status updates. ## ✅ next steps - [user guide](/docs/user-guide/) — everything the app can do - [addon development](/docs/addon-development/) — extend gguf loader with addons
  • common issues and their fixes. running the app from a terminal (`python main.py` or the installed binary) shows the console log, which is the fastest way to diagnose most problems. ## 🚀 app won't start ### windows smartscreen blocks the .exe - click **more info → run anyway**. the binaries are unsigned open-source builds; this is expected. ### "llama-cpp-python is required but not installed" - the llama.cpp runtime is missing. in the app's **environment** section click **install missing dependencies**, or from source: `pip install -r requirements.txt`. ### dll load errors on windows (`importerror: dll load failed`) - the bundled `llama_cpp/lib` wasn't found. make sure you're running the full package (not just copying a single file), and check the console log for `added windows dll directory: ...`. ## 🧠 model won't load ### "failed to load model" / unknown format - confirm the file is a real gguf file (`.gguf`). corrupted or partial downloads fail; re-download the file. - check the file has enough disk space and isn't on a network drive with strict permissions. ### out of memory / crashes when loading - reduce **context length** (try 4096 or 8192). - switch **processing** to `cpu only` if gpu mode crashes (vram exhaustion). - use a smaller quantized model (q4_0/q4_k_m instead of q8). ### gpu accelerated is slow or doesn't use the gpu - gpu mode requires a cuda-enabled `llama-cpp-python` build. verify with `scripts/verify_gpu_support.py` or run `scripts/install_gpu_llama.bat`/`.sh` (see the [installation guide](/docs/installation/)). - on linux, make sure the nvidia driver and cuda toolkit are installed. ## 💬 chat problems ### send button is disabled - no model is loaded. load a `.gguf` model first. ### responses are empty or garbled - some models need specific prompt formats. check the model card for a chat template; very small models also produce weak output. - lower `temperature`-style randomness isn't exposed per-chat — try a different preset or model. ### chat is slow - smaller context, cpu-only on fast threads, and a smaller quantized model all help. generation speed is dominated by model size and hardware. ## 🤖 agent mode issues ### "please load a model first" - agent mode needs a loaded model — load one, then toggle agent mode on. ### agent can't find files / "path escapes workspace" - the agent is **sandboxed to the workspace folder**. put files you want it to touch inside the workspace, or choose the right folder with the 📁 button. ### agent stuck on "processing..." - generation is single-threaded; long tool chains take time. watch the status messages for progress. ## 💬 floating chat addon ### button doesn't float above other apps on linux - this is expected under **wayland** — compositors confine app windows to the app itself. run under an x11 session or with `qt_qpa_platform=xcb` for full always-on-top behavior. ### button disappears when switching apps on macos - fixed in 2.1.2 (the `tool` flag is now dropped on macos so the button stays visible). if it still happens, check the console log and confirm the addon is listed under **addons**. ### button is stuck minimized - fixed in 2.1.2 — the button now restores itself immediately if a system shortcut minimizes it. ### addon not in the addons menu - click **addons → refresh addons**, or restart the app. check the console for `failed to load addon ...`. ## 🪟 window / layout problems ### window too small / layout broken - the minimum window size is 800×500. maximize or resize the splitter — the sidebar is collapsible via the splitter handle. ### wrong colors / theme looks off - **view → dark mode** toggles themes. if colors look broken after a theme change, restart the app. ## 🐍 environment / venv issues ### "not running from .venv" - the app detected the system python. click **create .venv & restart** in the sidebar to bootstrap an isolated environment. ### pip install fails with externally-managed-environment (linux) - on debian/ubuntu 24.04+ the system python blocks pip. use the app's **create .venv & restart**, or create a venv manually: ```bash python3 -m venv .venv && source .venv/bin/activate && pip install -r requirements.txt ``` ## 📁 where to find logs - **windows**: `%localappdata%\ggufloader\logs` - **linux/macos**: `~/.ggufloader/logs` include the relevant log excerpt when [opening an issue](https://github.com/ggufloader/gguf-loader/issues).
  • this is the complete manual for gguf loader 2.1.2. gguf loader is a privacy-first desktop app for running large language models locally from gguf files, with zero data leaving your machine. ## 🪟 the main window the app is divided into three areas: 1. **header bar** (top) — brand on the left, a live **model status chip** on the right. 2. **sidebar** (left) — model settings, environment, and launcher sections. 3. **chat area** (center) — the conversation, or a welcome screen when no model is loaded. ## ⚙️ model settings sidebar ### model - **load gguf model** — opens a file dialog; pick any `.gguf` file. - **model info** — shows the loaded file name (or an error). ### processing - **cpu only** — runs on any machine using llama.cpp's cpu backend. - **gpu accelerated** — offloads layers to an nvidia gpu (windows/linux). requires a working cuda llama-cpp-python install. ### context length the model's context window in tokens (512–32768). larger contexts use more ram. change it before loading a model. ## 📊 environment section gguf loader ships with a built-in dependency manager for source installs: - **python · .venv status** — shows the interpreter and whether the app runs from a virtual environment. - **install missing dependencies** — appears when packages are missing; runs `pip install -r requirements.txt`. - **create .venv & restart** — bootstraps a virtual environment and relaunches inside it. - **check again** — re-scans the environment. > the packaged installers (`.exe` / linux tarball) bundle everything, so this section is mostly relevant when running from source. ## 🚀 launcher section one-click buttons that open the project's `scripts/` utilities (e.g. gpu support verification, monitor) in separate windows, plus **restart app**. ## 💬 chatting - type in the input box; **enter** sends, **shift+enter** inserts a newline. - **send** is disabled until you type something, and is disabled entirely until a model is loaded. - responses **stream** token-by-token into bubbles: your messages right (amber), ai left (charcoal). - **view → text size** (12–22) changes bubble font size live. - **file → clear chat** wipes the conversation (the model stays loaded). ## 🤖 agent mode agent mode turns the chat into a tool-using assistant that works inside a **workspace folder**: 1. toggle **🤖 agent mode: off → on**. 2. choose a workspace (combo box or 📁 browse button). default: `./agent_workspace`. 3. ask for file operations — e.g. *"create a markdown file listing today's tasks"*. the agent will: - analyze complex requests, - plan tool calls and stream status updates (🤔/💡/→/✓/✗), - execute tools against the workspace — `list_directory`, `read_file`, `write_file`, `edit_file`, `search_files`, - summarize results in natural language. **safety**: all tools are sandboxed to the workspace; paths that escape it are rejected. ## 🎨 appearance - **view → dark mode** toggles the "midnight & amber" dark theme and a light theme. - **view → text size** adjusts chat bubble text. - the app remembers dark mode per session (dark is the default). ## 🧩 addons the **addons** menu lists every loaded addon (e.g. **floating_chat**). selecting one opens it in a floating dialog; **refresh addons** re-scans the `addons/` folder. see the [addon development guide](/docs/addon-development/). ### 💬 floating chat the built-in addon adds a messenger-style floating button (always on top, draggable) that opens a chat window connected to the loaded model. it: - stays on top of all windows (see platform notes below) - remembers its position between sessions - shows model status (🟢 ready / 🔴 offline) and streams responses - has copy all / clear controls **platform notes**: fully floating on windows and linux/x11. on linux **wayland**, compositors confine it to the app window — run under x11 (`qt_qpa_platform=xcb`) for the full effect. on macos the button stays visible when the app loses focus but also appears in mission control. ## 📁 where files live - **config**: `%appdata%\ggufloader` (windows) / `~/.ggufloader` (linux/macos) - **cache**: `%localappdata%\ggufloader\cache` (windows) / `~/.cache/ggufloader` (linux) - **logs**: `%localappdata%\ggufloader\logs` (windows) / `~/.ggufloader/logs` (linux) - **addons**: the `addons/` folder next to the app ## ❤️ feedback **help → send feedback** opens the feedback dialog. point it at your own formspree endpoint via `feedback_config.json`: ```json { "endpoint_url": "https://formspree.io/f/your_form_id" } ``` ## 🆘 still stuck? see the [troubleshooting guide](/docs/troubleshooting/).

  🔗 related features

  📥

  download center get gguf loader for your platform

  🤖

  model library find the perfect model for your needs

  🎯 what's next?

  you've completed this guide! here are some suggested next steps to continue your gguf loader journey:

  🚀

  start using gguf loader

  now that you have gguf loader installed, follow our quick start guide to load your first model and start chatting.

  quick start guide →

  🤖

  browse models

  explore our curated collection of gguf models to find the perfect one for your needs.

  browse models →

  ✨

  explore features

  discover all the powerful features that make gguf loader special.

  see features →

  🌟 explore more features

  interactive demos model library community projects user stories

  🏠 back to homepage

  ✨

  explore features see what gguf loader can do

  📥

  download now get gguf loader for your system

  🤖

  browse models find the perfect model for your needs

  👥

  join community connect with other users
• troubleshooting guide

  solutions to common gguf loader issues and problems

  beginner 10 minutes

  common issues and their fixes. running the app from a terminal (python main.py or the installed binary) shows the console log, which is the fastest way to diagnose most problems.

  🚀 app won’t start

  windows smartscreen blocks the .exe

  • click more info → run anyway. the binaries are unsigned open-source builds; this is expected.

  “llama-cpp-python is required but not installed”

  • the llama.cpp runtime is missing. in the app’s environment section click install missing dependencies, or from source: pip install -r requirements.txt.

  dll load errors on windows (importerror: dll load failed)

  • the bundled llama_cpp/lib wasn’t found. make sure you’re running the full package (not just copying a single file), and check the console log for added windows dll directory: ....

  🧠 model won’t load

  “failed to load model” / unknown format

  • confirm the file is a real gguf file (.gguf). corrupted or partial downloads fail; re-download the file.
  • check the file has enough disk space and isn’t on a network drive with strict permissions.

  out of memory / crashes when loading

  • reduce context length (try 4096 or 8192).
  • switch processing to cpu only if gpu mode crashes (vram exhaustion).
  • use a smaller quantized model (q4_0/q4_k_m instead of q8).

  gpu accelerated is slow or doesn’t use the gpu

  • gpu mode requires a cuda-enabled llama-cpp-python build. verify with scripts/verify_gpu_support.py or run scripts/install_gpu_llama.bat/.sh (see the installation guide).
  • on linux, make sure the nvidia driver and cuda toolkit are installed.

  💬 chat problems

  send button is disabled

  • no model is loaded. load a .gguf model first.

  responses are empty or garbled

  • some models need specific prompt formats. check the model card for a chat template; very small models also produce weak output.
  • lower temperature-style randomness isn’t exposed per-chat — try a different preset or model.

  chat is slow

  • smaller context, cpu-only on fast threads, and a smaller quantized model all help. generation speed is dominated by model size and hardware.

  🤖 agent mode issues

  “please load a model first”

  • agent mode needs a loaded model — load one, then toggle agent mode on.

  agent can’t find files / “path escapes workspace”

  • the agent is sandboxed to the workspace folder. put files you want it to touch inside the workspace, or choose the right folder with the 📁 button.

  agent stuck on “processing…”

  • generation is single-threaded; long tool chains take time. watch the status messages for progress.

  💬 floating chat addon

  button doesn’t float above other apps on linux

  • this is expected under wayland — compositors confine app windows to the app itself. run under an x11 session or with qt_qpa_platform=xcb for full always-on-top behavior.

  button disappears when switching apps on macos

  • fixed in 2.1.2 (the tool flag is now dropped on macos so the button stays visible). if it still happens, check the console log and confirm the addon is listed under addons.

  button is stuck minimized

  • fixed in 2.1.2 — the button now restores itself immediately if a system shortcut minimizes it.

  addon not in the addons menu

  • click addons → refresh addons, or restart the app. check the console for failed to load addon ....

  🪟 window / layout problems

  window too small / layout broken

  • the minimum window size is 800×500. maximize or resize the splitter — the sidebar is collapsible via the splitter handle.

  wrong colors / theme looks off

  • view → dark mode toggles themes. if colors look broken after a theme change, restart the app.

  🐍 environment / venv issues

  “not running from .venv”

  • the app detected the system python. click create .venv & restart in the sidebar to bootstrap an isolated environment.

  pip install fails with externally-managed-environment (linux)

  • on debian/ubuntu 24.04+ the system python blocks pip. use the app’s create .venv & restart, or create a venv manually:

    python3 -m venv .venv && source .venv/bin/activate && pip install -r requirements.txt
    

  📁 where to find logs

  • windows: %localappdata%\ggufloader\logs
  • linux/macos: ~/.ggufloader/logs

  include the relevant log excerpt when opening an issue.

  related documentation

  • installation guide

    complete guide to installing gguf loader on windows, macos, and linux

    beginner 5 minutes

    this guide will help you install gguf loader 2.2.0 on your system. want to see what gguf loader can do first? explore the features on our homepage.

    📋 system requirements

    • os: windows 10/11, linux (x86_64), or macos
    • ram: 4 gb minimum (8 gb+ recommended for larger models)
    • storage: 2 gb free space for the app, plus room for model files
    • gpu: optional — cpu-only works everywhere; nvidia cuda is supported on windows and linux
    • python: not required for the installers (everything is bundled)

    🚀 quick installation

    the prebuilt installers bundle python, pyside6, and llama.cpp — you do not need python installed.

    windows

    1. download ggufloader_v2.2.0_cpu.exe (~69 mb) from the releases page. this is the cpu-only build that works on any pc.
    2. have an nvidia gpu with cuda? grab ggufloader_v2.2.0_gpu.exe (~854 mb) instead for gpu acceleration.
    3. double-click the downloaded file and run it. no installation wizard — the app starts immediately.

    ⚠️ windows smartscreen: the first launch may show “windows protected your pc”. click more info → run anyway. this is normal for unsigned open-source binaries.

    linux (x86_64)

    1. download ggufloader_v2.2.0_linux_x86_64_cpu (~121 mb) from the releases page.
    2. make it executable and run:

       chmod +x ggufloader_v2.2.0_linux_x86_64_cpu
       ./ggufloader_v2.2.0_linux_x86_64_cpu
       

       no installation needed — the app starts directly. everything (python, pyside6, llama.cpp) is bundled.

    macos

    no prebuilt installer is published yet. run from source (below) — everything works on macos.

    run from source (any platform)

    for development, or to run on an unsupported platform:

    git clone https://github.com/ggufloader/gguf-loader.git
    cd gguf-loader
    pip install -r requirements.txt
    python main.py
    

    ✅ verify your installation

    run ggufloader_v2.2.0_cpu.exe --version (windows) or ./ggufloader_v2.2.0_linux_x86_64_cpu --version (linux) — it prints gguf loader version 2.2.0 and exits.

    🔧 next steps

    • quick start guide — load your first model
    • troubleshooting — if something isn’t working

    related documentation

    • get up and running with gguf loader 2.2.0 in just a few minutes! new to gguf loader? [check out the homepage](/) to see what makes it special. ## 🚀 step 1: install gguf loader follow the [installation guide](/docs/installation/) for your platform, or download directly: - **windows**: [ggufloader_v2.2.0_cpu.exe](https://github.com/ggufloader/gguf-loader/releases/download/v2.2.0/ggufloader_v2.2.0_cpu.exe) (cpu-only) · [ggufloader_v2.2.0_gpu.exe](https://github.com/ggufloader/gguf-loader/releases/download/v2.2.0/ggufloader_v2.2.0_gpu.exe) (nvidia cuda) - **linux**: [ggufloader_v2.2.0_linux_x86_64_cpu](https://github.com/ggufloader/gguf-loader/releases/download/v2.2.0/ggufloader_v2.2.0_linux_x86_64_cpu) ## 📥 step 2: get a gguf model gguf loader runs any gguf-format model. good starter models on hugging face: - **small (fast)**: [llama-3-8b-instruct gguf (q4_0)](https://huggingface.co/thebloke/llama-3-8b-instruct-gguf) — ~4.7 gb, runs great on cpu - **medium**: [mistral-7b-instruct gguf (q4_k_m)](https://huggingface.co/thebloke/mistral-7b-instruct-v0.2-gguf) — ~4.1 gb - **larger**: [phi-3.5-mini-instruct gguf](https://huggingface.co/microsoft/phi-3.5-mini-instruct-gguf) or any qwen/llama gguf you like download the `.gguf` file and remember where you saved it. ## 🖥️ step 3: load the model 1. launch gguf loader. 2. in the **model settings** sidebar, choose **processing** mode — `cpu only` (default, works everywhere) or `gpu accelerated` (nvidia cuda). 3. set a **context length** (8192 is a safe default for most models; 32768 uses more ram). 4. click **load gguf model** and select your `.gguf` file. 5. wait for the status to show **"model ready!"** — the header chip turns green with the model name. ## 💬 step 4: chat type a message in the input box at the bottom and press **enter** (shift+enter inserts a newline). responses stream in as chatgpt-style bubbles — your messages on the right (amber), the ai's on the left. use the **view → text size** menu to adjust bubble font size. ## 🤖 step 5 (optional): try agent mode click **🤖 agent mode: off** in the input area to toggle it on: 1. pick a **workspace folder** (defaults to `./agent_workspace`). 2. ask the agent to do file work — e.g. *"create a file called hello.py that prints 'hi'"*. 3. the agent plans tool calls, executes them (read/write/edit/search files inside the workspace only), and reports back with live status updates. ## ✅ next steps - [user guide](/docs/user-guide/) — everything the app can do - [addon development](/docs/addon-development/) — extend gguf loader with addons
    • common issues and their fixes. running the app from a terminal (`python main.py` or the installed binary) shows the console log, which is the fastest way to diagnose most problems. ## 🚀 app won't start ### windows smartscreen blocks the .exe - click **more info → run anyway**. the binaries are unsigned open-source builds; this is expected. ### "llama-cpp-python is required but not installed" - the llama.cpp runtime is missing. in the app's **environment** section click **install missing dependencies**, or from source: `pip install -r requirements.txt`. ### dll load errors on windows (`importerror: dll load failed`) - the bundled `llama_cpp/lib` wasn't found. make sure you're running the full package (not just copying a single file), and check the console log for `added windows dll directory: ...`. ## 🧠 model won't load ### "failed to load model" / unknown format - confirm the file is a real gguf file (`.gguf`). corrupted or partial downloads fail; re-download the file. - check the file has enough disk space and isn't on a network drive with strict permissions. ### out of memory / crashes when loading - reduce **context length** (try 4096 or 8192). - switch **processing** to `cpu only` if gpu mode crashes (vram exhaustion). - use a smaller quantized model (q4_0/q4_k_m instead of q8). ### gpu accelerated is slow or doesn't use the gpu - gpu mode requires a cuda-enabled `llama-cpp-python` build. verify with `scripts/verify_gpu_support.py` or run `scripts/install_gpu_llama.bat`/`.sh` (see the [installation guide](/docs/installation/)). - on linux, make sure the nvidia driver and cuda toolkit are installed. ## 💬 chat problems ### send button is disabled - no model is loaded. load a `.gguf` model first. ### responses are empty or garbled - some models need specific prompt formats. check the model card for a chat template; very small models also produce weak output. - lower `temperature`-style randomness isn't exposed per-chat — try a different preset or model. ### chat is slow - smaller context, cpu-only on fast threads, and a smaller quantized model all help. generation speed is dominated by model size and hardware. ## 🤖 agent mode issues ### "please load a model first" - agent mode needs a loaded model — load one, then toggle agent mode on. ### agent can't find files / "path escapes workspace" - the agent is **sandboxed to the workspace folder**. put files you want it to touch inside the workspace, or choose the right folder with the 📁 button. ### agent stuck on "processing..." - generation is single-threaded; long tool chains take time. watch the status messages for progress. ## 💬 floating chat addon ### button doesn't float above other apps on linux - this is expected under **wayland** — compositors confine app windows to the app itself. run under an x11 session or with `qt_qpa_platform=xcb` for full always-on-top behavior. ### button disappears when switching apps on macos - fixed in 2.1.2 (the `tool` flag is now dropped on macos so the button stays visible). if it still happens, check the console log and confirm the addon is listed under **addons**. ### button is stuck minimized - fixed in 2.1.2 — the button now restores itself immediately if a system shortcut minimizes it. ### addon not in the addons menu - click **addons → refresh addons**, or restart the app. check the console for `failed to load addon ...`. ## 🪟 window / layout problems ### window too small / layout broken - the minimum window size is 800×500. maximize or resize the splitter — the sidebar is collapsible via the splitter handle. ### wrong colors / theme looks off - **view → dark mode** toggles themes. if colors look broken after a theme change, restart the app. ## 🐍 environment / venv issues ### "not running from .venv" - the app detected the system python. click **create .venv & restart** in the sidebar to bootstrap an isolated environment. ### pip install fails with externally-managed-environment (linux) - on debian/ubuntu 24.04+ the system python blocks pip. use the app's **create .venv & restart**, or create a venv manually: ```bash python3 -m venv .venv && source .venv/bin/activate && pip install -r requirements.txt ``` ## 📁 where to find logs - **windows**: `%localappdata%\ggufloader\logs` - **linux/macos**: `~/.ggufloader/logs` include the relevant log excerpt when [opening an issue](https://github.com/ggufloader/gguf-loader/issues).
    • this is the complete manual for gguf loader 2.1.2. gguf loader is a privacy-first desktop app for running large language models locally from gguf files, with zero data leaving your machine. ## 🪟 the main window the app is divided into three areas: 1. **header bar** (top) — brand on the left, a live **model status chip** on the right. 2. **sidebar** (left) — model settings, environment, and launcher sections. 3. **chat area** (center) — the conversation, or a welcome screen when no model is loaded. ## ⚙️ model settings sidebar ### model - **load gguf model** — opens a file dialog; pick any `.gguf` file. - **model info** — shows the loaded file name (or an error). ### processing - **cpu only** — runs on any machine using llama.cpp's cpu backend. - **gpu accelerated** — offloads layers to an nvidia gpu (windows/linux). requires a working cuda llama-cpp-python install. ### context length the model's context window in tokens (512–32768). larger contexts use more ram. change it before loading a model. ## 📊 environment section gguf loader ships with a built-in dependency manager for source installs: - **python · .venv status** — shows the interpreter and whether the app runs from a virtual environment. - **install missing dependencies** — appears when packages are missing; runs `pip install -r requirements.txt`. - **create .venv & restart** — bootstraps a virtual environment and relaunches inside it. - **check again** — re-scans the environment. > the packaged installers (`.exe` / linux tarball) bundle everything, so this section is mostly relevant when running from source. ## 🚀 launcher section one-click buttons that open the project's `scripts/` utilities (e.g. gpu support verification, monitor) in separate windows, plus **restart app**. ## 💬 chatting - type in the input box; **enter** sends, **shift+enter** inserts a newline. - **send** is disabled until you type something, and is disabled entirely until a model is loaded. - responses **stream** token-by-token into bubbles: your messages right (amber), ai left (charcoal). - **view → text size** (12–22) changes bubble font size live. - **file → clear chat** wipes the conversation (the model stays loaded). ## 🤖 agent mode agent mode turns the chat into a tool-using assistant that works inside a **workspace folder**: 1. toggle **🤖 agent mode: off → on**. 2. choose a workspace (combo box or 📁 browse button). default: `./agent_workspace`. 3. ask for file operations — e.g. *"create a markdown file listing today's tasks"*. the agent will: - analyze complex requests, - plan tool calls and stream status updates (🤔/💡/→/✓/✗), - execute tools against the workspace — `list_directory`, `read_file`, `write_file`, `edit_file`, `search_files`, - summarize results in natural language. **safety**: all tools are sandboxed to the workspace; paths that escape it are rejected. ## 🎨 appearance - **view → dark mode** toggles the "midnight & amber" dark theme and a light theme. - **view → text size** adjusts chat bubble text. - the app remembers dark mode per session (dark is the default). ## 🧩 addons the **addons** menu lists every loaded addon (e.g. **floating_chat**). selecting one opens it in a floating dialog; **refresh addons** re-scans the `addons/` folder. see the [addon development guide](/docs/addon-development/). ### 💬 floating chat the built-in addon adds a messenger-style floating button (always on top, draggable) that opens a chat window connected to the loaded model. it: - stays on top of all windows (see platform notes below) - remembers its position between sessions - shows model status (🟢 ready / 🔴 offline) and streams responses - has copy all / clear controls **platform notes**: fully floating on windows and linux/x11. on linux **wayland**, compositors confine it to the app window — run under x11 (`qt_qpa_platform=xcb`) for the full effect. on macos the button stays visible when the app loses focus but also appears in mission control. ## 📁 where files live - **config**: `%appdata%\ggufloader` (windows) / `~/.ggufloader` (linux/macos) - **cache**: `%localappdata%\ggufloader\cache` (windows) / `~/.cache/ggufloader` (linux) - **logs**: `%localappdata%\ggufloader\logs` (windows) / `~/.ggufloader/logs` (linux) - **addons**: the `addons/` folder next to the app ## ❤️ feedback **help → send feedback** opens the feedback dialog. point it at your own formspree endpoint via `feedback_config.json`: ```json { "endpoint_url": "https://formspree.io/f/your_form_id" } ``` ## 🆘 still stuck? see the [troubleshooting guide](/docs/troubleshooting/).

    🔗 related features

    📥

    download center get gguf loader for your platform

    🤖

    model library find the perfect model for your needs

    🎯 what's next?

    you've completed this guide! here are some suggested next steps to continue your gguf loader journey:

    🚀

    start using gguf loader

    now that you have gguf loader installed, follow our quick start guide to load your first model and start chatting.

    quick start guide →

    🤖

    browse models

    explore our curated collection of gguf models to find the perfect one for your needs.

    browse models →

    ✨

    explore features

    discover all the powerful features that make gguf loader special.

    see features →

    🌟 explore more features

    interactive demos model library community projects user stories

    🏠 back to homepage

    ✨

    explore features see what gguf loader can do

    📥

    download now get gguf loader for your system

    🤖

    browse models find the perfect model for your needs

    👥

    join community connect with other users
  • this is the complete manual for gguf loader 2.1.2. gguf loader is a privacy-first desktop app for running large language models locally from gguf files, with zero data leaving your machine. ## 🪟 the main window the app is divided into three areas: 1. **header bar** (top) — brand on the left, a live **model status chip** on the right. 2. **sidebar** (left) — model settings, environment, and launcher sections. 3. **chat area** (center) — the conversation, or a welcome screen when no model is loaded. ## ⚙️ model settings sidebar ### model - **load gguf model** — opens a file dialog; pick any `.gguf` file. - **model info** — shows the loaded file name (or an error). ### processing - **cpu only** — runs on any machine using llama.cpp's cpu backend. - **gpu accelerated** — offloads layers to an nvidia gpu (windows/linux). requires a working cuda llama-cpp-python install. ### context length the model's context window in tokens (512–32768). larger contexts use more ram. change it before loading a model. ## 📊 environment section gguf loader ships with a built-in dependency manager for source installs: - **python · .venv status** — shows the interpreter and whether the app runs from a virtual environment. - **install missing dependencies** — appears when packages are missing; runs `pip install -r requirements.txt`. - **create .venv & restart** — bootstraps a virtual environment and relaunches inside it. - **check again** — re-scans the environment. > the packaged installers (`.exe` / linux tarball) bundle everything, so this section is mostly relevant when running from source. ## 🚀 launcher section one-click buttons that open the project's `scripts/` utilities (e.g. gpu support verification, monitor) in separate windows, plus **restart app**. ## 💬 chatting - type in the input box; **enter** sends, **shift+enter** inserts a newline. - **send** is disabled until you type something, and is disabled entirely until a model is loaded. - responses **stream** token-by-token into bubbles: your messages right (amber), ai left (charcoal). - **view → text size** (12–22) changes bubble font size live. - **file → clear chat** wipes the conversation (the model stays loaded). ## 🤖 agent mode agent mode turns the chat into a tool-using assistant that works inside a **workspace folder**: 1. toggle **🤖 agent mode: off → on**. 2. choose a workspace (combo box or 📁 browse button). default: `./agent_workspace`. 3. ask for file operations — e.g. *"create a markdown file listing today's tasks"*. the agent will: - analyze complex requests, - plan tool calls and stream status updates (🤔/💡/→/✓/✗), - execute tools against the workspace — `list_directory`, `read_file`, `write_file`, `edit_file`, `search_files`, - summarize results in natural language. **safety**: all tools are sandboxed to the workspace; paths that escape it are rejected. ## 🎨 appearance - **view → dark mode** toggles the "midnight & amber" dark theme and a light theme. - **view → text size** adjusts chat bubble text. - the app remembers dark mode per session (dark is the default). ## 🧩 addons the **addons** menu lists every loaded addon (e.g. **floating_chat**). selecting one opens it in a floating dialog; **refresh addons** re-scans the `addons/` folder. see the [addon development guide](/docs/addon-development/). ### 💬 floating chat the built-in addon adds a messenger-style floating button (always on top, draggable) that opens a chat window connected to the loaded model. it: - stays on top of all windows (see platform notes below) - remembers its position between sessions - shows model status (🟢 ready / 🔴 offline) and streams responses - has copy all / clear controls **platform notes**: fully floating on windows and linux/x11. on linux **wayland**, compositors confine it to the app window — run under x11 (`qt_qpa_platform=xcb`) for the full effect. on macos the button stays visible when the app loses focus but also appears in mission control. ## 📁 where files live - **config**: `%appdata%\ggufloader` (windows) / `~/.ggufloader` (linux/macos) - **cache**: `%localappdata%\ggufloader\cache` (windows) / `~/.cache/ggufloader` (linux) - **logs**: `%localappdata%\ggufloader\logs` (windows) / `~/.ggufloader/logs` (linux) - **addons**: the `addons/` folder next to the app ## ❤️ feedback **help → send feedback** opens the feedback dialog. point it at your own formspree endpoint via `feedback_config.json`: ```json { "endpoint_url": "https://formspree.io/f/your_form_id" } ``` ## 🆘 still stuck? see the [troubleshooting guide](/docs/troubleshooting/).
  • quick start guide

    get up and running with gguf loader in just a few minutes

    beginner 5 minutes

    get up and running with gguf loader 2.2.0 in just a few minutes! new to gguf loader? check out the homepage to see what makes it special.

    🚀 step 1: install gguf loader

    follow the installation guide for your platform, or download directly:

    • windows: ggufloader_v2.2.0_cpu.exe (cpu-only) · ggufloader_v2.2.0_gpu.exe (nvidia cuda)
    • linux: ggufloader_v2.2.0_linux_x86_64_cpu

    📥 step 2: get a gguf model

    gguf loader runs any gguf-format model. good starter models on hugging face:

    • small (fast): llama-3-8b-instruct gguf (q4_0) — ~4.7 gb, runs great on cpu
    • medium: mistral-7b-instruct gguf (q4_k_m) — ~4.1 gb
    • larger: phi-3.5-mini-instruct gguf or any qwen/llama gguf you like

    download the .gguf file and remember where you saved it.

    🖥️ step 3: load the model

    1. launch gguf loader.
    2. in the model settings sidebar, choose processing mode — cpu only (default, works everywhere) or gpu accelerated (nvidia cuda).
    3. set a context length (8192 is a safe default for most models; 32768 uses more ram).
    4. click load gguf model and select your .gguf file.
    5. wait for the status to show “model ready!” — the header chip turns green with the model name.

    💬 step 4: chat

    type a message in the input box at the bottom and press enter (shift+enter inserts a newline). responses stream in as chatgpt-style bubbles — your messages on the right (amber), the ai’s on the left.

    use the view → text size menu to adjust bubble font size.

    🤖 step 5 (optional): try agent mode

    click 🤖 agent mode: off in the input area to toggle it on:

    1. pick a workspace folder (defaults to ./agent_workspace).
    2. ask the agent to do file work — e.g. “create a file called hello.py that prints ‘hi’“.
    3. the agent plans tool calls, executes them (read/write/edit/search files inside the workspace only), and reports back with live status updates.

    ✅ next steps

    • user guide — everything the app can do
    • addon development — extend gguf loader with addons

    related documentation

    • installation guide

      complete guide to installing gguf loader on windows, macos, and linux

      beginner 5 minutes

      this guide will help you install gguf loader 2.2.0 on your system. want to see what gguf loader can do first? explore the features on our homepage.

      📋 system requirements

      • os: windows 10/11, linux (x86_64), or macos
      • ram: 4 gb minimum (8 gb+ recommended for larger models)
      • storage: 2 gb free space for the app, plus room for model files
      • gpu: optional — cpu-only works everywhere; nvidia cuda is supported on windows and linux
      • python: not required for the installers (everything is bundled)

      🚀 quick installation

      the prebuilt installers bundle python, pyside6, and llama.cpp — you do not need python installed.

      windows

      1. download ggufloader_v2.2.0_cpu.exe (~69 mb) from the releases page. this is the cpu-only build that works on any pc.
      2. have an nvidia gpu with cuda? grab ggufloader_v2.2.0_gpu.exe (~854 mb) instead for gpu acceleration.
      3. double-click the downloaded file and run it. no installation wizard — the app starts immediately.

      ⚠️ windows smartscreen: the first launch may show “windows protected your pc”. click more info → run anyway. this is normal for unsigned open-source binaries.

      linux (x86_64)

      1. download ggufloader_v2.2.0_linux_x86_64_cpu (~121 mb) from the releases page.
      2. make it executable and run:

         chmod +x ggufloader_v2.2.0_linux_x86_64_cpu
         ./ggufloader_v2.2.0_linux_x86_64_cpu
         

         no installation needed — the app starts directly. everything (python, pyside6, llama.cpp) is bundled.

      macos

      no prebuilt installer is published yet. run from source (below) — everything works on macos.

      run from source (any platform)

      for development, or to run on an unsupported platform:

      git clone https://github.com/ggufloader/gguf-loader.git
      cd gguf-loader
      pip install -r requirements.txt
      python main.py
      

      ✅ verify your installation

      run ggufloader_v2.2.0_cpu.exe --version (windows) or ./ggufloader_v2.2.0_linux_x86_64_cpu --version (linux) — it prints gguf loader version 2.2.0 and exits.

      🔧 next steps

      • quick start guide — load your first model
      • troubleshooting — if something isn’t working

      related documentation

      • get up and running with gguf loader 2.2.0 in just a few minutes! new to gguf loader? [check out the homepage](/) to see what makes it special. ## 🚀 step 1: install gguf loader follow the [installation guide](/docs/installation/) for your platform, or download directly: - **windows**: [ggufloader_v2.2.0_cpu.exe](https://github.com/ggufloader/gguf-loader/releases/download/v2.2.0/ggufloader_v2.2.0_cpu.exe) (cpu-only) · [ggufloader_v2.2.0_gpu.exe](https://github.com/ggufloader/gguf-loader/releases/download/v2.2.0/ggufloader_v2.2.0_gpu.exe) (nvidia cuda) - **linux**: [ggufloader_v2.2.0_linux_x86_64_cpu](https://github.com/ggufloader/gguf-loader/releases/download/v2.2.0/ggufloader_v2.2.0_linux_x86_64_cpu) ## 📥 step 2: get a gguf model gguf loader runs any gguf-format model. good starter models on hugging face: - **small (fast)**: [llama-3-8b-instruct gguf (q4_0)](https://huggingface.co/thebloke/llama-3-8b-instruct-gguf) — ~4.7 gb, runs great on cpu - **medium**: [mistral-7b-instruct gguf (q4_k_m)](https://huggingface.co/thebloke/mistral-7b-instruct-v0.2-gguf) — ~4.1 gb - **larger**: [phi-3.5-mini-instruct gguf](https://huggingface.co/microsoft/phi-3.5-mini-instruct-gguf) or any qwen/llama gguf you like download the `.gguf` file and remember where you saved it. ## 🖥️ step 3: load the model 1. launch gguf loader. 2. in the **model settings** sidebar, choose **processing** mode — `cpu only` (default, works everywhere) or `gpu accelerated` (nvidia cuda). 3. set a **context length** (8192 is a safe default for most models; 32768 uses more ram). 4. click **load gguf model** and select your `.gguf` file. 5. wait for the status to show **"model ready!"** — the header chip turns green with the model name. ## 💬 step 4: chat type a message in the input box at the bottom and press **enter** (shift+enter inserts a newline). responses stream in as chatgpt-style bubbles — your messages on the right (amber), the ai's on the left. use the **view → text size** menu to adjust bubble font size. ## 🤖 step 5 (optional): try agent mode click **🤖 agent mode: off** in the input area to toggle it on: 1. pick a **workspace folder** (defaults to `./agent_workspace`). 2. ask the agent to do file work — e.g. *"create a file called hello.py that prints 'hi'"*. 3. the agent plans tool calls, executes them (read/write/edit/search files inside the workspace only), and reports back with live status updates. ## ✅ next steps - [user guide](/docs/user-guide/) — everything the app can do - [addon development](/docs/addon-development/) — extend gguf loader with addons
      • common issues and their fixes. running the app from a terminal (`python main.py` or the installed binary) shows the console log, which is the fastest way to diagnose most problems. ## 🚀 app won't start ### windows smartscreen blocks the .exe - click **more info → run anyway**. the binaries are unsigned open-source builds; this is expected. ### "llama-cpp-python is required but not installed" - the llama.cpp runtime is missing. in the app's **environment** section click **install missing dependencies**, or from source: `pip install -r requirements.txt`. ### dll load errors on windows (`importerror: dll load failed`) - the bundled `llama_cpp/lib` wasn't found. make sure you're running the full package (not just copying a single file), and check the console log for `added windows dll directory: ...`. ## 🧠 model won't load ### "failed to load model" / unknown format - confirm the file is a real gguf file (`.gguf`). corrupted or partial downloads fail; re-download the file. - check the file has enough disk space and isn't on a network drive with strict permissions. ### out of memory / crashes when loading - reduce **context length** (try 4096 or 8192). - switch **processing** to `cpu only` if gpu mode crashes (vram exhaustion). - use a smaller quantized model (q4_0/q4_k_m instead of q8). ### gpu accelerated is slow or doesn't use the gpu - gpu mode requires a cuda-enabled `llama-cpp-python` build. verify with `scripts/verify_gpu_support.py` or run `scripts/install_gpu_llama.bat`/`.sh` (see the [installation guide](/docs/installation/)). - on linux, make sure the nvidia driver and cuda toolkit are installed. ## 💬 chat problems ### send button is disabled - no model is loaded. load a `.gguf` model first. ### responses are empty or garbled - some models need specific prompt formats. check the model card for a chat template; very small models also produce weak output. - lower `temperature`-style randomness isn't exposed per-chat — try a different preset or model. ### chat is slow - smaller context, cpu-only on fast threads, and a smaller quantized model all help. generation speed is dominated by model size and hardware. ## 🤖 agent mode issues ### "please load a model first" - agent mode needs a loaded model — load one, then toggle agent mode on. ### agent can't find files / "path escapes workspace" - the agent is **sandboxed to the workspace folder**. put files you want it to touch inside the workspace, or choose the right folder with the 📁 button. ### agent stuck on "processing..." - generation is single-threaded; long tool chains take time. watch the status messages for progress. ## 💬 floating chat addon ### button doesn't float above other apps on linux - this is expected under **wayland** — compositors confine app windows to the app itself. run under an x11 session or with `qt_qpa_platform=xcb` for full always-on-top behavior. ### button disappears when switching apps on macos - fixed in 2.1.2 (the `tool` flag is now dropped on macos so the button stays visible). if it still happens, check the console log and confirm the addon is listed under **addons**. ### button is stuck minimized - fixed in 2.1.2 — the button now restores itself immediately if a system shortcut minimizes it. ### addon not in the addons menu - click **addons → refresh addons**, or restart the app. check the console for `failed to load addon ...`. ## 🪟 window / layout problems ### window too small / layout broken - the minimum window size is 800×500. maximize or resize the splitter — the sidebar is collapsible via the splitter handle. ### wrong colors / theme looks off - **view → dark mode** toggles themes. if colors look broken after a theme change, restart the app. ## 🐍 environment / venv issues ### "not running from .venv" - the app detected the system python. click **create .venv & restart** in the sidebar to bootstrap an isolated environment. ### pip install fails with externally-managed-environment (linux) - on debian/ubuntu 24.04+ the system python blocks pip. use the app's **create .venv & restart**, or create a venv manually: ```bash python3 -m venv .venv && source .venv/bin/activate && pip install -r requirements.txt ``` ## 📁 where to find logs - **windows**: `%localappdata%\ggufloader\logs` - **linux/macos**: `~/.ggufloader/logs` include the relevant log excerpt when [opening an issue](https://github.com/ggufloader/gguf-loader/issues).
      • this is the complete manual for gguf loader 2.1.2. gguf loader is a privacy-first desktop app for running large language models locally from gguf files, with zero data leaving your machine. ## 🪟 the main window the app is divided into three areas: 1. **header bar** (top) — brand on the left, a live **model status chip** on the right. 2. **sidebar** (left) — model settings, environment, and launcher sections. 3. **chat area** (center) — the conversation, or a welcome screen when no model is loaded. ## ⚙️ model settings sidebar ### model - **load gguf model** — opens a file dialog; pick any `.gguf` file. - **model info** — shows the loaded file name (or an error). ### processing - **cpu only** — runs on any machine using llama.cpp's cpu backend. - **gpu accelerated** — offloads layers to an nvidia gpu (windows/linux). requires a working cuda llama-cpp-python install. ### context length the model's context window in tokens (512–32768). larger contexts use more ram. change it before loading a model. ## 📊 environment section gguf loader ships with a built-in dependency manager for source installs: - **python · .venv status** — shows the interpreter and whether the app runs from a virtual environment. - **install missing dependencies** — appears when packages are missing; runs `pip install -r requirements.txt`. - **create .venv & restart** — bootstraps a virtual environment and relaunches inside it. - **check again** — re-scans the environment. > the packaged installers (`.exe` / linux tarball) bundle everything, so this section is mostly relevant when running from source. ## 🚀 launcher section one-click buttons that open the project's `scripts/` utilities (e.g. gpu support verification, monitor) in separate windows, plus **restart app**. ## 💬 chatting - type in the input box; **enter** sends, **shift+enter** inserts a newline. - **send** is disabled until you type something, and is disabled entirely until a model is loaded. - responses **stream** token-by-token into bubbles: your messages right (amber), ai left (charcoal). - **view → text size** (12–22) changes bubble font size live. - **file → clear chat** wipes the conversation (the model stays loaded). ## 🤖 agent mode agent mode turns the chat into a tool-using assistant that works inside a **workspace folder**: 1. toggle **🤖 agent mode: off → on**. 2. choose a workspace (combo box or 📁 browse button). default: `./agent_workspace`. 3. ask for file operations — e.g. *"create a markdown file listing today's tasks"*. the agent will: - analyze complex requests, - plan tool calls and stream status updates (🤔/💡/→/✓/✗), - execute tools against the workspace — `list_directory`, `read_file`, `write_file`, `edit_file`, `search_files`, - summarize results in natural language. **safety**: all tools are sandboxed to the workspace; paths that escape it are rejected. ## 🎨 appearance - **view → dark mode** toggles the "midnight & amber" dark theme and a light theme. - **view → text size** adjusts chat bubble text. - the app remembers dark mode per session (dark is the default). ## 🧩 addons the **addons** menu lists every loaded addon (e.g. **floating_chat**). selecting one opens it in a floating dialog; **refresh addons** re-scans the `addons/` folder. see the [addon development guide](/docs/addon-development/). ### 💬 floating chat the built-in addon adds a messenger-style floating button (always on top, draggable) that opens a chat window connected to the loaded model. it: - stays on top of all windows (see platform notes below) - remembers its position between sessions - shows model status (🟢 ready / 🔴 offline) and streams responses - has copy all / clear controls **platform notes**: fully floating on windows and linux/x11. on linux **wayland**, compositors confine it to the app window — run under x11 (`qt_qpa_platform=xcb`) for the full effect. on macos the button stays visible when the app loses focus but also appears in mission control. ## 📁 where files live - **config**: `%appdata%\ggufloader` (windows) / `~/.ggufloader` (linux/macos) - **cache**: `%localappdata%\ggufloader\cache` (windows) / `~/.cache/ggufloader` (linux) - **logs**: `%localappdata%\ggufloader\logs` (windows) / `~/.ggufloader/logs` (linux) - **addons**: the `addons/` folder next to the app ## ❤️ feedback **help → send feedback** opens the feedback dialog. point it at your own formspree endpoint via `feedback_config.json`: ```json { "endpoint_url": "https://formspree.io/f/your_form_id" } ``` ## 🆘 still stuck? see the [troubleshooting guide](/docs/troubleshooting/).

      🔗 related features

      📥

      download center get gguf loader for your platform

      🤖

      model library find the perfect model for your needs

      🎯 what's next?

      you've completed this guide! here are some suggested next steps to continue your gguf loader journey:

      🚀

      start using gguf loader

      now that you have gguf loader installed, follow our quick start guide to load your first model and start chatting.

      quick start guide →

      🤖

      browse models

      explore our curated collection of gguf models to find the perfect one for your needs.

      browse models →

      ✨

      explore features

      discover all the powerful features that make gguf loader special.

      see features →

      🌟 explore more features

      interactive demos model library community projects user stories

      🏠 back to homepage

      ✨

      explore features see what gguf loader can do

      📥

      download now get gguf loader for your system

      🤖

      browse models find the perfect model for your needs

      👥

      join community connect with other users
    • this is the complete manual for gguf loader 2.1.2. gguf loader is a privacy-first desktop app for running large language models locally from gguf files, with zero data leaving your machine. ## 🪟 the main window the app is divided into three areas: 1. **header bar** (top) — brand on the left, a live **model status chip** on the right. 2. **sidebar** (left) — model settings, environment, and launcher sections. 3. **chat area** (center) — the conversation, or a welcome screen when no model is loaded. ## ⚙️ model settings sidebar ### model - **load gguf model** — opens a file dialog; pick any `.gguf` file. - **model info** — shows the loaded file name (or an error). ### processing - **cpu only** — runs on any machine using llama.cpp's cpu backend. - **gpu accelerated** — offloads layers to an nvidia gpu (windows/linux). requires a working cuda llama-cpp-python install. ### context length the model's context window in tokens (512–32768). larger contexts use more ram. change it before loading a model. ## 📊 environment section gguf loader ships with a built-in dependency manager for source installs: - **python · .venv status** — shows the interpreter and whether the app runs from a virtual environment. - **install missing dependencies** — appears when packages are missing; runs `pip install -r requirements.txt`. - **create .venv & restart** — bootstraps a virtual environment and relaunches inside it. - **check again** — re-scans the environment. > the packaged installers (`.exe` / linux tarball) bundle everything, so this section is mostly relevant when running from source. ## 🚀 launcher section one-click buttons that open the project's `scripts/` utilities (e.g. gpu support verification, monitor) in separate windows, plus **restart app**. ## 💬 chatting - type in the input box; **enter** sends, **shift+enter** inserts a newline. - **send** is disabled until you type something, and is disabled entirely until a model is loaded. - responses **stream** token-by-token into bubbles: your messages right (amber), ai left (charcoal). - **view → text size** (12–22) changes bubble font size live. - **file → clear chat** wipes the conversation (the model stays loaded). ## 🤖 agent mode agent mode turns the chat into a tool-using assistant that works inside a **workspace folder**: 1. toggle **🤖 agent mode: off → on**. 2. choose a workspace (combo box or 📁 browse button). default: `./agent_workspace`. 3. ask for file operations — e.g. *"create a markdown file listing today's tasks"*. the agent will: - analyze complex requests, - plan tool calls and stream status updates (🤔/💡/→/✓/✗), - execute tools against the workspace — `list_directory`, `read_file`, `write_file`, `edit_file`, `search_files`, - summarize results in natural language. **safety**: all tools are sandboxed to the workspace; paths that escape it are rejected. ## 🎨 appearance - **view → dark mode** toggles the "midnight & amber" dark theme and a light theme. - **view → text size** adjusts chat bubble text. - the app remembers dark mode per session (dark is the default). ## 🧩 addons the **addons** menu lists every loaded addon (e.g. **floating_chat**). selecting one opens it in a floating dialog; **refresh addons** re-scans the `addons/` folder. see the [addon development guide](/docs/addon-development/). ### 💬 floating chat the built-in addon adds a messenger-style floating button (always on top, draggable) that opens a chat window connected to the loaded model. it: - stays on top of all windows (see platform notes below) - remembers its position between sessions - shows model status (🟢 ready / 🔴 offline) and streams responses - has copy all / clear controls **platform notes**: fully floating on windows and linux/x11. on linux **wayland**, compositors confine it to the app window — run under x11 (`qt_qpa_platform=xcb`) for the full effect. on macos the button stays visible when the app loses focus but also appears in mission control. ## 📁 where files live - **config**: `%appdata%\ggufloader` (windows) / `~/.ggufloader` (linux/macos) - **cache**: `%localappdata%\ggufloader\cache` (windows) / `~/.cache/ggufloader` (linux) - **logs**: `%localappdata%\ggufloader\logs` (windows) / `~/.ggufloader/logs` (linux) - **addons**: the `addons/` folder next to the app ## ❤️ feedback **help → send feedback** opens the feedback dialog. point it at your own formspree endpoint via `feedback_config.json`: ```json { "endpoint_url": "https://formspree.io/f/your_form_id" } ``` ## 🆘 still stuck? see the [troubleshooting guide](/docs/troubleshooting/).
    • addon development guide

      learn to create custom addons for gguf loader with examples and best practices

      advanced 15 minutes

      gguf loader’s addon system lets you extend the app without touching its source. addons are python packages dropped into the addons/ folder; the app discovers, loads, and manages them automatically.

      📦 what an addon looks like

      addons/
      └── my_addon/
          ├── __init__.py      # must expose register()
          ├── main.py          # your logic (any structure you like)
          └── ...              # widgets, resources, etc.
      

      minimum viable addon

      # addons/my_addon/__init__.py
      from pyside6.qtwidgets import qlabel
      
      def register(parent=none):
          """called by gguf loader when the addon loads."""
          return qlabel("hello from my addon!")
      

      that’s it — drop the folder in addons/, restart (or addons → refresh addons), and your widget appears in the addons menu.

      🔍 how loading works

      addonmanager (addon_manager.py):

      1. scan_addons() — lists subdirectories of the addons folder that contain __init__.py.
      2. load_addon(name, path) — dynamically imports the module (importlib.util.spec_from_file_location) and requires a register attribute.
      3. load_all_addons() — loads every addon and records success/failure in the console log.
      4. get_addon_widget(name, parent) — calls register(parent) and returns the widget.
      5. open_addon_dialog(name, parent) — shows the addon widget in a non-modal dialog from the addons menu.

      the main window calls get_addon_widget for each loaded addon at startup, so register() runs even before the user opens the menu — this is how background addons (like the floating chat button) start automatically.

      📡 talking to the app

      your register(parent) receives the parent widget. the main window exposes an addon-facing api (see the addon api reference). the robust pattern used by the built-in addon:

      def register(parent=none):
          gguf_app = none
          # 1. parent might already be the main window
          if parent and hasattr(parent, 'model') and hasattr(parent, 'model_loaded'):
              gguf_app = parent
          else:
              # 2. walk up the parent chain
              current = parent
              while current is not none:
                  if hasattr(current, 'model') and hasattr(current, 'model_loaded'):
                      gguf_app = current
                      break
                  current = current.parent()
              # 3. fall back to scanning top-level widgets
              if gguf_app is none:
                  app = qapplication.instance()
                  for widget in app.toplevelwidgets():
                      if hasattr(widget, 'model') and hasattr(widget, 'model_loaded'):
                          gguf_app = widget
                          break
          ...
      

      key api members: model (callable backend), model_loaded / model_unloaded / generation_finished / generation_error signals, addon_manager, and _floating_chat_addon.

      🤖 calling the model

      # streaming
      for token in gguf_app.model(prompt, stream=true, max_tokens=512):
          print(token["choices"][0]["text"])
      
      # or use the backend directly
      response = gguf_app.model.generate(prompt, max_tokens=512, temperature=0.7)
      

      model is none until the user loads a model — always check, and subscribe to model_loaded to react.

      🧩 widget vs. background addons

      • return a widget from register() → appears in the addons menu, opens in a dialog.
      • return none (or also keep your own top-level windows) → runs in the background. the floating chat addon returns a small status widget but drives its own always-on-top windows.

      💡 best practices

      1. guard everything — wrap registration in try/except and log failures; a broken addon must never crash the app.
      2. clean up state — if you keep references on the main window (e.g. gguf_app._my_addon = ...), stop them on app close (mainwindow.closeevent stops _floating_chat_addon — follow that pattern).
      3. use qsettings for persistence — qsettings("ggufloader", "youraddon").
      4. prefer signals — let your widgets emit signals; don’t reach into other widgets’ internals.
      5. qt-free logic — if you have non-ui logic (parsing, computation), put it in a module with no qt imports so it’s unit-testable.

      ✅ debugging

      addon load errors print to the console with a traceback (failed to load addon <name>: ...). run the app from a terminal to see them:

      python main.py
      

      📚 next steps

      • addon api reference — every hook and member
      • floating chat example — full walkthrough of the built-in addon

      related documentation

      • addon api reference

        complete api reference for developing gguf loader addons

        advanced 20 minutes

        complete reference for the hooks, signals, and objects available to gguf loader addons.

        🪝 addon contract

        an addon is a folder under addons/ with an __init__.py exposing:

        register(parent=none) -> qwidget | none

        called by addonmanager when the addon loads. returns an optional widget shown in the addons menu (opened in a non-modal dialog), or none for background-only addons.

        parameters — parent: the widget the addon is being attached to (may be the main window, a dialog, or another widget).

        return — a widget, or none.

        def register(parent=none):
            return qlabel("my addon")   # widget addon
        

        🖥️ main window api

        addons receive a reference to the main window (via parent or by scanning qapplication.instance().toplevelwidgets()). it exposes:

        properties

        | member | type | description | |—|—|—| | model | modelbackend \| none | the loaded model backend; callable. none until a model is loaded. | | chat_generator | none | legacy hook; always none. addons should call model instead. | | addon_manager | addonmanager | the addon manager instance. | | _floating_chat_addon | floatingchataddon \| none | the built-in floating chat addon (if running). |

        signals

        | signal | signature | description | |—|—|—| | model_loaded | signal(object) | emitted with the modelbackend after a model loads. | | model_unloaded | signal() | emitted after the model is released. | | generation_finished | signal() | emitted when a chat generation completes. | | generation_error | signal(str) | emitted with an error message when generation fails. | | theme_changed | signal(bool) | emitted when dark mode toggles (true = dark). |

        gguf_app.model_loaded.connect(self._on_model_loaded)
        gguf_app.generation_finished.connect(self._on_finished)
        

        🧠 modelbackend

        gguf_app.model is a core.llm.model_backend.modelbackend. it is callable and thread-safe (all access serialized through a lock).

        calling convention

        # raw llama-cpp shape (addon-compatible)
        result = gguf_app.model(prompt, stream=false, max_tokens=512, temperature=0.7)
        text = result["choices"][0]["text"]
        
        # streaming
        for token_data in gguf_app.model(prompt, stream=true, max_tokens=512):
            text = token_data["choices"][0]["text"]
        

        methods & attributes

        | member | description | |—|—| | model_path | path of the loaded gguf file. | | use_gpu | whether gpu acceleration is active. | | n_ctx | context length. | | is_loaded | true once the runtime exists. | | load() | create the underlying llama runtime (raises on failure). | | unload() | release the model and free memory. | | generate(prompt, **kwargs) | complete non-streamed response string. | | generate_stream(prompt, **kwargs) | iterator of token strings. | | __call__(prompt, **kwargs) | llama-cpp-compatible callable (stream returns token-dict generator). |

        kwargs are passed straight to llama_cpp: max_tokens, temperature, top_p, top_k, repeat_penalty, stop, etc.

        🧰 agent tools (for agent-capable addons)

        the agent engine (core/agent/) exposes sandboxed tools through toolregistry. tools are not qt-bound and are safe to call from any thread.

        tool	params	result
        list_directory	path (default .)	{status, result: [{name, type, size}]}
        read_file	path, max_size, encoding	{status, result: content, lines, encoding}
        write_file	path, content	{status, path, bytes_written}
        edit_file	path, operation (replace/insert_line/delete_line), find, replace, line_number, content	{status, operation, changes_made}
        search_files	pattern/query, path	{status, result: [paths], total_matches}

        every tool result includes status ("success" / "error") and tool_name. all paths are sandboxed to the workspace — escaping paths raise.

        from core.agent.tool_registry import create_default_registry
        registry = create_default_registry("./workspace")
        result = registry.execute("list_directory", {"path": "."})
        

        🧩 addonmanager

        method	description
        scan_addons()	{name: path} of addons with __init__.py.
        load_addon(name, path)	import + validate register; true/false.
        load_all_addons()	load all; {name: success}.
        get_addon_widget(name, parent)	widget from register(parent).
        open_addon_dialog(name, parent)	show addon in a non-modal dialog.
        get_loaded_addons()	list of successfully loaded names.

        💾 persistence

        use qt’s qsettings with a scoped org/app key to avoid collisions:

        from pyside6.qtcore import qsettings
        settings = qsettings("ggufloader", "myaddon")
        settings.setvalue("position", point)
        pos = settings.value("position", qpoint(100, 100))
        

        ⚡ lifecycle

        • register() is called at startup and on addons → refresh addons.
        • the main window’s closeevent stops the floating chat addon; follow that pattern to stop your own background objects:

          if hasattr(gguf_app, '_my_addon') and gguf_app._my_addon:
              gguf_app._my_addon.stop()
          

        📚 see also

        • addon development guide — tutorial
        • floating chat example — full real-world addon

        related documentation

        • gguf loader's addon system lets you extend the app without touching its source. addons are python packages dropped into the `addons/` folder; the app discovers, loads, and manages them automatically. ## 📦 what an addon looks like ``` addons/ └── my_addon/ ├── __init__.py # must expose register() ├── main.py # your logic (any structure you like) └── ... # widgets, resources, etc. ``` ### minimum viable addon ```python # addons/my_addon/__init__.py from pyside6.qtwidgets import qlabel def register(parent=none): """called by gguf loader when the addon loads.""" return qlabel("hello from my addon!") ``` that's it — drop the folder in `addons/`, restart (or **addons → refresh addons**), and your widget appears in the **addons** menu. ## 🔍 how loading works `addonmanager` (`addon_manager.py`): 1. `scan_addons()` — lists subdirectories of the addons folder that contain `__init__.py`. 2. `load_addon(name, path)` — dynamically imports the module (`importlib.util.spec_from_file_location`) and requires a `register` attribute. 3. `load_all_addons()` — loads every addon and records success/failure in the console log. 4. `get_addon_widget(name, parent)` — calls `register(parent)` and returns the widget. 5. `open_addon_dialog(name, parent)` — shows the addon widget in a non-modal dialog from the addons menu. the main window calls `get_addon_widget` for each loaded addon at startup, so `register()` runs even before the user opens the menu — this is how background addons (like the floating chat button) start automatically. ## 📡 talking to the app your `register(parent)` receives the parent widget. the **main window** exposes an addon-facing api (see the [addon api reference](/docs/addon-api/)). the robust pattern used by the built-in addon: ```python def register(parent=none): gguf_app = none # 1. parent might already be the main window if parent and hasattr(parent, 'model') and hasattr(parent, 'model_loaded'): gguf_app = parent else: # 2. walk up the parent chain current = parent while current is not none: if hasattr(current, 'model') and hasattr(current, 'model_loaded'): gguf_app = current break current = current.parent() # 3. fall back to scanning top-level widgets if gguf_app is none: app = qapplication.instance() for widget in app.toplevelwidgets(): if hasattr(widget, 'model') and hasattr(widget, 'model_loaded'): gguf_app = widget break ... ``` key api members: `model` (callable backend), `model_loaded` / `model_unloaded` / `generation_finished` / `generation_error` signals, `addon_manager`, and `_floating_chat_addon`. ## 🤖 calling the model ```python # streaming for token in gguf_app.model(prompt, stream=true, max_tokens=512): print(token["choices"][0]["text"]) # or use the backend directly response = gguf_app.model.generate(prompt, max_tokens=512, temperature=0.7) ``` `model` is `none` until the user loads a model — always check, and subscribe to `model_loaded` to react. ## 🧩 widget vs. background addons - **return a widget** from `register()` → appears in the addons menu, opens in a dialog. - **return `none`** (or also keep your own top-level windows) → runs in the background. the floating chat addon returns a small status widget but drives its own always-on-top windows. ## 💡 best practices 1. **guard everything** — wrap registration in `try/except` and log failures; a broken addon must never crash the app. 2. **clean up state** — if you keep references on the main window (e.g. `gguf_app._my_addon = ...`), stop them on app close (`mainwindow.closeevent` stops `_floating_chat_addon` — follow that pattern). 3. **use qsettings for persistence** — `qsettings("ggufloader", "youraddon")`. 4. **prefer signals** — let your widgets emit signals; don't reach into other widgets' internals. 5. **qt-free logic** — if you have non-ui logic (parsing, computation), put it in a module with no qt imports so it's unit-testable. ## ✅ debugging addon load errors print to the console with a traceback (`failed to load addon : ...`). run the app from a terminal to see them: ```bash python main.py ``` ## 📚 next steps - [addon api reference](/docs/addon-api/) — every hook and member - [floating chat example](/docs/floating-chat-example/) — full walkthrough of the built-in addon
        • this page walks through **floating_chat**, gguf loader's built-in addon — a facebook messenger-style floating button that opens a chat window connected to the loaded model. it's the best real-world reference for writing a background addon with its own windows. ``` addons/floating_chat/ ├── __init__.py # exposes register() ├── main.py # floatingchataddon (qobject) + register() ├── floating_button.py # floatingchatbutton - draggable always-on-top button ├── chat_window.py # floatingchatwindow - the chat ui └── status_widget.py # small sidebar widget returned by register() ``` ## 🧠 the design the addon is a `qobject` (`floatingchataddon`) that owns two windows: - **`floatingchatbutton`** — a frameless, always-on-top tool window that can be dragged anywhere and remembers its position via `qsettings`. - **`floatingchatwindow`** — a regular top-level window (`400×600`) positioned next to the button, with a message list, streaming replies, and copy all / clear buttons. ## 🪝 registration `__init__.py` simply re-exports `register`: ```python from .main import register __all__ = ['register'] ``` `register()` finds the main window (parent → parent chain → `toplevelwidgets()` scan), stops any previous instance, starts the addon, and returns a **status widget** for the sidebar: ```python def register(parent=none): gguf_app = _find_main_window(parent) # model + model_loaded check if gguf_app is none: return none if getattr(gguf_app, '_floating_chat_addon', none): gguf_app._floating_chat_addon.stop() addon = floatingchataddon(gguf_app) if addon.start(): gguf_app._floating_chat_addon = addon return floatingchatstatuswidget(addon) return none ``` the returned `floatingchatstatuswidget` shows up in the **addons** menu, and because `register()` runs at startup, the button appears automatically — no user action needed. ## 📡 connecting to the app `floatingchataddon.__init__` subscribes to model lifecycle signals: ```python if hasattr(self.gguf_app, 'model_loaded'): self.gguf_app.model_loaded.connect(self._on_model_loaded) if hasattr(self.gguf_app, 'generation_finished'): self.gguf_app.generation_finished.connect(self._on_generation_finished) if hasattr(self.gguf_app, 'generation_error'): self.gguf_app.generation_error.connect(self._on_generation_error) ``` `_on_model_loaded` flips the chat window's status to **🟢 model: ready**; generation events update the streaming bubble / error state. ## 🏃 lifecycle: start() / stop() ```python def start(self): self._floating_button = floatingchatbutton() self._floating_button.clicked.connect(self._on_button_clicked) self._load_button_position() # qsettings restore, clamped to screen self._floating_button.show() self.addon_started.emit() return true def stop(self): self._save_button_position() # persist for next session if self._chat_window: self._chat_window.close() if self._floating_button: self._floating_button.close() self.addon_stopped.emit() ``` `stop()` is idempotent and called on app close via `mainwindow.closeevent`, so there's never a dangling button. ## 🖱️ button click → window toggle the click handler distinguishes three states (a minimized window is "visible" to qt but off-screen, so it must be restored, not hidden): ```python def _on_button_clicked(self): chat = self._chat_window if chat and chat.isvisible(): if chat.isminimized(): chat.shownormal(); chat.raise_(); chat.activatewindow() else: chat.hide() else: self._show_chat_window() ``` `_show_chat_window` creates the window lazily, positions it next to the button (flipping to the left / up when it would overflow the screen's **available geometry**, which excludes taskbars and docks), then `shownormal()` + `raise_()` + `activatewindow()`. ## 🪟 window flags (the platform subtleties) ```python flags = (qt.windowtype.framelesswindowhint | qt.windowtype.windowstaysontophint | qt.windowtype.x11bypasswindowmanagerhint) # linux/x11 if sys.platform != "darwin": flags |= qt.windowtype.tool # keeps it out of the taskbar ``` - **windows/linux**: `tool` keeps the button out of the taskbar. `x11bypasswindowmanagerhint` gives true always-on-top under x11. - **macos**: the `tool` flag would turn the button into a utility window that **auto-hides when the app loses focus** — so it's deliberately dropped. trade-off: the button appears in mission control. - **wayland**: compositors don't allow floating above *other* apps' windows; the button is confined to the app window (documented, and x11 is recommended for the full experience). the chat window drops the minimize button entirely — a minimized companion window is a trap state that can't be reliably restored. ## 📌 resiliancy touches - `changeevent` bounces the button straight back to normal if a system shortcut (e.g. win+d) minimizes *every* window including the tool: ```python if event.type() == qevent.type.windowstatechange and self.isminimized(): qtimer.singleshot(0, self.shownormal) ``` - dragging clamps to `screen.availablegeometry()` (not `(0,0)`), so the button can't slide under the macos menu bar or a windows taskbar. - saved positions are clamped the same way on restore. ## 🧩 what you can reuse - the **window-finding pattern** in `register()` — works for any addon. - the **qsettings position persistence** with on-screen clamping. - the **platform-conditional window flags** — copy it for any always-on-top addon window. - the **show/hide/restore toggle** logic. ## 📚 see also - [addon development guide](/docs/addon-development/) - [addon api reference](/docs/addon-api/)
        • this page documents how the gguf loader codebase is organized. the app is a pyside6 desktop application; a single `main.py` bootstraps everything, and logic is split into `core/` (pure logic), `services/` (qt threading bridges), and `ui/` + `widgets/` (presentation). ## 🗂️ top-level layout ``` gguf-loader/ ├── main.py # entry point: logging, dll paths, qapplication, mainwindow ├── config.py # central configuration constants ├── resource_manager.py # resource/path discovery (dev, package, or frozen) ├── addon_manager.py # loads and manages addons ├── requirements.txt # python dependencies ├── build_exe.spec # pyinstaller spec used for windows & linux builds │ ├── core/ # pure, testable logic (no qt) │ ├── llm/ │ │ ├── model_backend.py # thread-safe llama-cpp-python wrapper │ │ └── prompt_builder.py # system-prompt & conversation assembly │ └── agent/ │ ├── agent_engine.py # tool-use agent loop (no qt, no llama_cpp) │ └── tool_registry.py # sandboxed workspace tools │ ├── services/ # qobject bridges that run work on threads │ ├── model_service.py # load/unload models on a qthread │ ├── chat_service.py # streaming generation on a qthread │ ├── agent_service.py # runs agentengine on a worker thread │ ├── environment_service.py # venv/dependency checks & pip tasks │ └── launcher_service.py # launches scripts/ utilities │ ├── ui/ # main window & panels │ ├── main_window.py # composition root + addon-facing api │ ├── chat_panel.py # chat display, input, agent controls │ ├── sidebar_panel.py # model settings sidebar │ └── theme.py # dark/light qss token system │ ├── widgets/ # reusable widgets │ ├── chat_bubble.py # chatgpt-style bubble │ └── feedback_dialog.py # feedback form dialog │ ├── addons/ # addon packages (each has __init__.py with register()) │ └── floating_chat/ # built-in floating chat addon │ ├── scripts/ # utility & release scripts │ ├── capture_screenshots.py # regenerates readme/site screenshots │ ├── install_linux.sh # linux installer/uninstaller │ ├── package_linux.sh # builds the linux .tar.gz release │ └── ... # gpu install/monitor helpers │ └── build_hooks/ # pyinstaller hook modules ``` ## 🔁 key design rules - **`core/` never imports qt or llama_cpp.** it receives plain callables, so it can be unit-tested in isolation. - **`services/` are the only place qt threads are created.** ui never spins up threads directly. - **`ui/` widgets are "dumb"** — they render state and emit signals; `mainwindow` owns all logic. - **`resource_manager.py`** makes paths work identically in dev, as an installed package, and in a pyinstaller bundle (`sys._meipass`). ## 🧵 threading model ``` ui thread (mainwindow) worker thread │ │ │── modelservice.load() ─────→│ qthread: llama_cpp loads model │←────── loaded(modelbackend) │ │── chatservice.generate() ──→│ qthread: streams tokens │←────── token_received(text) │ │── agentservice.process() ──→│ qthread: agentengine tool loop │←────── status/tool/response │ ``` a fresh `qthread` + worker is created per request (the professional qt pattern — `qthread` is never subclassed). ## 🚀 entry point flow `main.py` → `setup_library_path()` (finds bundled llama.cpp libs) → `qapplication` → `mainwindow()` → builds ui, wires services, checks environment, loads addons → `app.exec()`. ## 📦 packaging - **windows**: `build_exe.bat` / `build_exe.spec` → `ggufloader_vx.y.z.exe` - **linux**: build the spec inside a linux environment, then `scripts/package_linux.sh` wraps the binary + installer + icon into a `.tar.gz` - github actions publishes both to every release automatically. see the [architecture overview](/docs/architecture/) for deeper design rationale.

        🔗 related features

        👥

        community hub connect with other developers

        🛠️

        addon showcase see community creations

        🎯 what's next?

        you've completed this guide! here are some suggested next steps to continue your gguf loader journey:

        🚀

        share your addon

        built something awesome? share it with the community and get featured on our homepage.

        share with community →

        📖

        study examples

        learn from real-world addon examples and see what's possible.

        addon showcase →

        🤝

        get support

        need help with your addon? our community is here to support you.

        get help →

        🌟 explore more features

        interactive demos model library community projects user stories

        🏠 back to homepage

        ✨

        explore features see what gguf loader can do

        📥

        download now get gguf loader for your system

        🤖

        browse models find the perfect model for your needs

        👥

        join community connect with other users
      • this page walks through **floating_chat**, gguf loader's built-in addon — a facebook messenger-style floating button that opens a chat window connected to the loaded model. it's the best real-world reference for writing a background addon with its own windows. ``` addons/floating_chat/ ├── __init__.py # exposes register() ├── main.py # floatingchataddon (qobject) + register() ├── floating_button.py # floatingchatbutton - draggable always-on-top button ├── chat_window.py # floatingchatwindow - the chat ui └── status_widget.py # small sidebar widget returned by register() ``` ## 🧠 the design the addon is a `qobject` (`floatingchataddon`) that owns two windows: - **`floatingchatbutton`** — a frameless, always-on-top tool window that can be dragged anywhere and remembers its position via `qsettings`. - **`floatingchatwindow`** — a regular top-level window (`400×600`) positioned next to the button, with a message list, streaming replies, and copy all / clear buttons. ## 🪝 registration `__init__.py` simply re-exports `register`: ```python from .main import register __all__ = ['register'] ``` `register()` finds the main window (parent → parent chain → `toplevelwidgets()` scan), stops any previous instance, starts the addon, and returns a **status widget** for the sidebar: ```python def register(parent=none): gguf_app = _find_main_window(parent) # model + model_loaded check if gguf_app is none: return none if getattr(gguf_app, '_floating_chat_addon', none): gguf_app._floating_chat_addon.stop() addon = floatingchataddon(gguf_app) if addon.start(): gguf_app._floating_chat_addon = addon return floatingchatstatuswidget(addon) return none ``` the returned `floatingchatstatuswidget` shows up in the **addons** menu, and because `register()` runs at startup, the button appears automatically — no user action needed. ## 📡 connecting to the app `floatingchataddon.__init__` subscribes to model lifecycle signals: ```python if hasattr(self.gguf_app, 'model_loaded'): self.gguf_app.model_loaded.connect(self._on_model_loaded) if hasattr(self.gguf_app, 'generation_finished'): self.gguf_app.generation_finished.connect(self._on_generation_finished) if hasattr(self.gguf_app, 'generation_error'): self.gguf_app.generation_error.connect(self._on_generation_error) ``` `_on_model_loaded` flips the chat window's status to **🟢 model: ready**; generation events update the streaming bubble / error state. ## 🏃 lifecycle: start() / stop() ```python def start(self): self._floating_button = floatingchatbutton() self._floating_button.clicked.connect(self._on_button_clicked) self._load_button_position() # qsettings restore, clamped to screen self._floating_button.show() self.addon_started.emit() return true def stop(self): self._save_button_position() # persist for next session if self._chat_window: self._chat_window.close() if self._floating_button: self._floating_button.close() self.addon_stopped.emit() ``` `stop()` is idempotent and called on app close via `mainwindow.closeevent`, so there's never a dangling button. ## 🖱️ button click → window toggle the click handler distinguishes three states (a minimized window is "visible" to qt but off-screen, so it must be restored, not hidden): ```python def _on_button_clicked(self): chat = self._chat_window if chat and chat.isvisible(): if chat.isminimized(): chat.shownormal(); chat.raise_(); chat.activatewindow() else: chat.hide() else: self._show_chat_window() ``` `_show_chat_window` creates the window lazily, positions it next to the button (flipping to the left / up when it would overflow the screen's **available geometry**, which excludes taskbars and docks), then `shownormal()` + `raise_()` + `activatewindow()`. ## 🪟 window flags (the platform subtleties) ```python flags = (qt.windowtype.framelesswindowhint | qt.windowtype.windowstaysontophint | qt.windowtype.x11bypasswindowmanagerhint) # linux/x11 if sys.platform != "darwin": flags |= qt.windowtype.tool # keeps it out of the taskbar ``` - **windows/linux**: `tool` keeps the button out of the taskbar. `x11bypasswindowmanagerhint` gives true always-on-top under x11. - **macos**: the `tool` flag would turn the button into a utility window that **auto-hides when the app loses focus** — so it's deliberately dropped. trade-off: the button appears in mission control. - **wayland**: compositors don't allow floating above *other* apps' windows; the button is confined to the app window (documented, and x11 is recommended for the full experience). the chat window drops the minimize button entirely — a minimized companion window is a trap state that can't be reliably restored. ## 📌 resiliancy touches - `changeevent` bounces the button straight back to normal if a system shortcut (e.g. win+d) minimizes *every* window including the tool: ```python if event.type() == qevent.type.windowstatechange and self.isminimized(): qtimer.singleshot(0, self.shownormal) ``` - dragging clamps to `screen.availablegeometry()` (not `(0,0)`), so the button can't slide under the macos menu bar or a windows taskbar. - saved positions are clamped the same way on restore. ## 🧩 what you can reuse - the **window-finding pattern** in `register()` — works for any addon. - the **qsettings position persistence** with on-screen clamping. - the **platform-conditional window flags** — copy it for any always-on-top addon window. - the **show/hide/restore toggle** logic. ## 📚 see also - [addon development guide](/docs/addon-development/) - [addon api reference](/docs/addon-api/)
      • this page documents how the gguf loader codebase is organized. the app is a pyside6 desktop application; a single `main.py` bootstraps everything, and logic is split into `core/` (pure logic), `services/` (qt threading bridges), and `ui/` + `widgets/` (presentation). ## 🗂️ top-level layout ``` gguf-loader/ ├── main.py # entry point: logging, dll paths, qapplication, mainwindow ├── config.py # central configuration constants ├── resource_manager.py # resource/path discovery (dev, package, or frozen) ├── addon_manager.py # loads and manages addons ├── requirements.txt # python dependencies ├── build_exe.spec # pyinstaller spec used for windows & linux builds │ ├── core/ # pure, testable logic (no qt) │ ├── llm/ │ │ ├── model_backend.py # thread-safe llama-cpp-python wrapper │ │ └── prompt_builder.py # system-prompt & conversation assembly │ └── agent/ │ ├── agent_engine.py # tool-use agent loop (no qt, no llama_cpp) │ └── tool_registry.py # sandboxed workspace tools │ ├── services/ # qobject bridges that run work on threads │ ├── model_service.py # load/unload models on a qthread │ ├── chat_service.py # streaming generation on a qthread │ ├── agent_service.py # runs agentengine on a worker thread │ ├── environment_service.py # venv/dependency checks & pip tasks │ └── launcher_service.py # launches scripts/ utilities │ ├── ui/ # main window & panels │ ├── main_window.py # composition root + addon-facing api │ ├── chat_panel.py # chat display, input, agent controls │ ├── sidebar_panel.py # model settings sidebar │ └── theme.py # dark/light qss token system │ ├── widgets/ # reusable widgets │ ├── chat_bubble.py # chatgpt-style bubble │ └── feedback_dialog.py # feedback form dialog │ ├── addons/ # addon packages (each has __init__.py with register()) │ └── floating_chat/ # built-in floating chat addon │ ├── scripts/ # utility & release scripts │ ├── capture_screenshots.py # regenerates readme/site screenshots │ ├── install_linux.sh # linux installer/uninstaller │ ├── package_linux.sh # builds the linux .tar.gz release │ └── ... # gpu install/monitor helpers │ └── build_hooks/ # pyinstaller hook modules ``` ## 🔁 key design rules - **`core/` never imports qt or llama_cpp.** it receives plain callables, so it can be unit-tested in isolation. - **`services/` are the only place qt threads are created.** ui never spins up threads directly. - **`ui/` widgets are "dumb"** — they render state and emit signals; `mainwindow` owns all logic. - **`resource_manager.py`** makes paths work identically in dev, as an installed package, and in a pyinstaller bundle (`sys._meipass`). ## 🧵 threading model ``` ui thread (mainwindow) worker thread │ │ │── modelservice.load() ─────→│ qthread: llama_cpp loads model │←────── loaded(modelbackend) │ │── chatservice.generate() ──→│ qthread: streams tokens │←────── token_received(text) │ │── agentservice.process() ──→│ qthread: agentengine tool loop │←────── status/tool/response │ ``` a fresh `qthread` + worker is created per request (the professional qt pattern — `qthread` is never subclassed). ## 🚀 entry point flow `main.py` → `setup_library_path()` (finds bundled llama.cpp libs) → `qapplication` → `mainwindow()` → builds ui, wires services, checks environment, loads addons → `app.exec()`. ## 📦 packaging - **windows**: `build_exe.bat` / `build_exe.spec` → `ggufloader_vx.y.z.exe` - **linux**: build the spec inside a linux environment, then `scripts/package_linux.sh` wraps the binary + installer + icon into a `.tar.gz` - github actions publishes both to every release automatically. see the [architecture overview](/docs/architecture/) for deeper design rationale.

      🔗 related features

      👥

      community hub connect with other developers

      🛠️

      addon showcase see community creations

      🎯 what's next?

      you've completed this guide! here are some suggested next steps to continue your gguf loader journey:

      🚀

      share your addon

      built something awesome? share it with the community and get featured on our homepage.

      share with community →

      📖

      study examples

      learn from real-world addon examples and see what's possible.

      addon showcase →

      🤝

      get support

      need help with your addon? our community is here to support you.

      get help →

      🌟 explore more features

      interactive demos model library community projects user stories

      🏠 back to homepage

      ✨

      explore features see what gguf loader can do

      📥

      download now get gguf loader for your system

      🤖

      browse models find the perfect model for your needs

      👥

      join community connect with other users

    🔗 related features

    ✨

    feature showcase see what gguf loader can do

    📚

    how-to guides learn with interactive tutorials

    🎯 what's next?

    you've completed this guide! here are some suggested next steps to continue your gguf loader journey:

    🛠️

    create your first addon

    ready to extend gguf loader? learn how to create custom addons with our development guide.

    addon development →

    👥

    join the community

    connect with other gguf loader users, share your projects, and get help.

    community hub →

    📚

    advanced tutorials

    take your skills to the next level with advanced how-to guides and tutorials.

    how-to guides →

    🌟 explore more features

    interactive demos model library community projects user stories

    🏠 back to homepage

    ✨

    explore features see what gguf loader can do

    📥

    download now get gguf loader for your system

    🤖

    browse models find the perfect model for your needs

    👥

    join community connect with other users

  🔗 related features

  ✨

  feature overview discover gguf loader's capabilities

  🚀

  get started download and start using gguf loader

  🎯 what's next?

  you've completed this guide! here are some suggested next steps to continue your gguf loader journey:

  🏠

  explore homepage

  discover more features, download options, and community resources on our homepage.

  visit homepage →

  📚

  more documentation

  continue learning with our comprehensive documentation library.

  all documentation →

  💬

  get support

  have questions? our community and support team are here to help.

  faq & support →

  🌟 explore more features

  interactive demos model library community projects user stories

  🏠 back to homepage

  ✨

  explore features see what gguf loader can do

  📥

  download now get gguf loader for your system

  🤖

  browse models find the perfect model for your needs

  👥

  join community connect with other users