Local AI Architecture Deep Dive / Part 3

AI Tools & Contextual Vision: How the AI "Sees" Your Web

Learn about the Recursive Tool-Calling framework and the "Browser Vision" system that allows the AI to perceive your current page content in real-time.

Contextual Awareness

1. Browser Vision: Capturing Context

To make the AI helpful, we provide it with a "snapshot" of what you are currently looking at. Every time you send a message, the frontend (arcade-chat.js or app-v3.js) performs a quick scrape of the current page.

// Frontend context capture logic
const pageContext = document.title;
const pageText = document.body.innerText.substring(0, 1000);
// This snippet is sent to the AI backend as "CURRENT CONTEXT"
            

This allows the AI to answer questions like "What am I looking at?" or "Explain the code on this page." even though it has no direct visual camera access.

Tool-Calling

2. Tool-Calling Tags

We use a custom "Tag Interception" system. The AI is trained via its system prompt to output specific brackets when it needs to perform an action. The Node.js server intercepts these brackets before the final response is shown.

[SEARCH: query]

[FETCH: url]

[OPEN: url]

[PLAY: action]

Search: Triggers a background scrape of DuckDuckGo Lite.
Fetch: Downloads the raw text content of any public URL.
Open: Launches a URI (like spotify:search) on your local OS.
Play: Sends playback commands (Play/Pause/Skip) to the media bridge.

Execution Loop

3. The Recursive Intelligence Loop

If the AI generates a [SEARCH] tag, the backend doesn't just show you the search result. It performs the search, appends the results to a new prompt, and re-calls the AI automatically.

// The Multi-Pass Execution Loop
if (hasToolTags) {
    const toolData = await executeWebTools(aiResponse);
    // Reprompt the AI with the new data
    aiResponse = await getChatResponse(originalMsg + "\n\nDATA: " + toolData);
}
            

This "Thinking" loop is what makes the Signal Share AI feel like it's actually browsing the web for you in real-time.

Back to Part 2 Next: Arcade Commands →