Keep the serving engine you need

Phone and edge SLMs are not Gemma-only

Small variants from Gemma, Qwen, Phi, SmolLM and Llama are candidate model families, not a universal phone support list. Check the exact model's license, quantization, context budget, runtime, available memory and thermal behavior on the target Android/iOS device. This project has not certified those combinations.

The Python openai-compatible worker accepts any model ID supported by its configured server; the job and worker IDs must match exactly. It can run on a desktop/gateway that calls a secured phone-hosted endpoint. It does not install a native phone server, and a browser page does not automatically expose HTTP. For example, after independently installing and testing your chosen server:

open-compute worker --backend openai-compatible --name edge-slm \
  --credentials edge-worker.json --model YOUR-SERVED-MODEL-ID \
  --inference-url https://YOUR-SECURED-DEVICE-ENDPOINT

The optional browser loader below is currently a Gemma-specific reference. The server-side adapters are model-neutral; that does not mean the browser loader already implements every SLM runtime or native mobile acceleration.

vLLM and SGLang

Both are serving choices through their OpenAI-compatible HTTP APIs. They are not installed by this project. Select versions compatible with your GPU, driver, model architecture and license; verify the endpoint before enrolling a worker.

open-compute worker --backend vllm --name gpu-01 \
  --credentials gpu-worker.json --model gemma \
  --inference-url http://127.0.0.1:8000

open-compute submit --kind chat \
  --payload '{"model":"gemma","prompt":"Explain a job lease.","max_tokens":64}'

Use --backend sglang for SGLang, or openai-compatible for another compatible server. The flag records backend identity; the wire protocol is intentionally shared. A worker claims only its exact chat:<model> capability. Its upstream URL and optional OPEN_COMPUTE_INFERENCE_KEY are configured locally, never supplied by a job. Network access and GPU capacity remain administrator-owned.

The NVIDIA/CUDA or AMD/ROCm runtime belongs on a supported Linux server. Neither the broker nor a phone creates PCIe bridges, installs drivers or configures IOMMU/VFIO passthrough. Model Fleet's Kubernetes device-plugin allocation is a separate deployment path, not automatic federation with this broker.

Browser Gemma

The browser UI has an explicit load Gemma with WebGPU action. It uses pinned WebLLM 0.2.86 and gemma-2-2b-it-q4f16_1-MLC-1k; the upstream config lists roughly 1.6 GB of model GPU memory, before browser/runtime overhead. Download size, available memory and browser feature support can still prevent loading. Respect the Gemma model license and third-party notices. This is an optional external model download, not code or weights bundled into this repository.

After loading, enroll/start the browser and submit chat with model gemma-browser. Stop terminates the browser compute worker, and reloading the model is necessary before advertising that capability again. The one-minute per-task budget can interrupt slow inference; use short prompts and outputs.

WebGPU availability is not proof of compatibility. No physical phone or browser GPU inference certification is claimed. CPU-only browsers can run the bounded proof demo, not Gemma CPU inference in this version.

Distributed does not mean shared memory

Jobs are distributed independently. This release does not shard a model over phones, train across unreliable Wi-Fi, or combine VRAM into one address space. Inference outputs are trusted-worker results, not mathematically verified answers.