Local Integration & System Architecture

1. Hardware Requirements

LLMs are "memory-bandwidth limited." The most important factor for performance is not raw compute, but how fast you can move the model's weights from memory to the processing unit.

The GPU Hierarchy:

• VRAM 8GB is the entry-point for small models (7B). 24GB (RTX 3090/4090) is the "sweet spot" for 30B-70B models.
• Unified Memory Apple Silicon (M1/M2/M3) allows the GPU to use regular RAM as VRAM, making it exceptionally powerful for large models.
• CPU Fallback If you lack a GPU, frameworks like llama.cpp can run models on your CPU, but at significantly lower speeds (tokens per second).

2. GGUF & Quantization

A full-precision model (FP16) requires ~2GB of RAM per billion parameters. Quantization compresses these weights from 16-bit to 4-bit or 8-bit, drastically reducing RAM usage with minimal intelligence loss.

GGUF is the standard file format for local inference. It contains both the model weights and the metadata required to run them, optimized for fast loading and cross-platform compatibility.

3. LM Studio: The Local Server

In the Signal Share ecosystem, LM Studio acts as an **Inference Provider**. It hosts the model and exposes an OpenAI-compatible API endpoint on your local network.

Server Configuration:

• Local Host: 127.0.0.1 (Ensures it's not accessible from the public internet).
• Port: Typically 12345 or 5000.
• Protocol: HTTP REST for completions, or WebSockets for streaming.

4. Implementing the Connection

To connect the Signal Share frontend to your local LLM, we use an asynchronous fetch pattern with a streaming response handler.

async function callLocalAI(prompt) {
  const response = await fetch("http://localhost:12345/v1/chat/completions", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      model: "llama-3-8b",
      messages: [{ role: "user", content: prompt }],
      stream: true // Enable real-time token rendering
    })
  });

  const reader = response.body.getReader();
  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    // Process stream chunks and update UI
  }
}

By using stream: true, we avoid "hanging" the UI while the model thinks. The user sees the text appearing word-by-word, mimicking the feel of a cloud-based assistant.