Newshot

Show HN: Shard – Distributed AI Inference Across Multiple Computers

Original text(via HN AI)

English | Русская версия | Қазақша нұсқасы

SHARD

One Network. Many Servers. Shared AI Infrastructure. Distributed local LLM inference engine pooling GPU VRAM and system RAM across LAN machines — powered by PyTorch, FastAPI, llama.cpp RPC, and native C++20 / MASM AVX2 SIMD kernels.

Latest Release
Latest Release
Tests CI
Tests CI

SHARD Distributed Inference Cluster: Central master coordinator pooling VRAM across connected LAN worker nodes over UDP discovery (:8001) and FastAPI SSE endpoints (:8000).

SHARD Hero Network Animation — Distributed AI Inference Cluster
SHARD Hero Network Animation — Distributed AI Inference Cluster

Overview · How It Works · Features · Quick Start · CLI Reference · Architecture · Dashboard · Community

What Is SHARD?

SHARD (shard-engine) is an open-source distributed local LLM inference engine written in Python (FastAPI + PyTorch) with a GGUF / llama.cpp backend and native C++20 / MASM AVX2 SIMD acceleration.

The Problem It Solves

Running capable 14B, 32B, or 70B+ open-weight language models locally typically hits a hard hardware wall: single-card VRAM limits. Most developers, labs, and small teams own multiple machines—a desktop gaming GPU, a workstation laptop, an auxiliary mini-PC—whose memory sits fragmented across the local network while no single device can fit the full model weights and KV cache.

Why SHARD Exists

SHARD unites idle GPUs and system memory across your Local Area Network (LAN) into a single logical inference cluster:

  • Pool Hardware Without Cloud Lock-In: Split transformer layers across multiple consumer GPUs and host RAM over standard gigabit LAN (lan.py, gguf_backend.py) with zero external cloud dependencies or recurring API fees.
  • Know Before You Download (shard fit): Inspect whether any Hugging Face model will fit your single GPU, hybrid CPU+GPU setup, or pooled LAN cluster before downloading gigabytes of weights (fit.py).
  • Drop-In Agent & IDE Backend: Expose both OpenAI (/v1/chat/completions, /v1/completions, /v1/models) and Anthropic (/v1/messages) compatible endpoints on port 8000 so coding agents and SDKs connect immediately.

Who Should Use SHARD?

  • Local AI Developers & Researchers running 7B–70B GGUF or PyTorch SafeTensors models across home lab or office workstations.
  • Software Engineering Teams hosting private, zero-telemetry coding assistants for Cursor, Claude Code, Aider, Continue, Zed, and OpenCode.
  • AI Agent Builders orchestrating multi-agent pipelines via OpenAI Python SDK, LangChain, or LlamaIndex against self-hosted hardware.

How SHARD Works

SHARD turns distributed local AI infrastructure into a four-step workflow: CREATE → SHARE → CONNECT → USE.

1. CREATE — Launch Your Server Start a local SHARD node with shard start or python main.py . The engine probes your CPU, system RAM, and NVIDIA CUDA / Apple MPS GPU via hardware.py , loads your chosen GGUF or Hugging Face model, and brings up the FastAPI server and Web UI on http://localhost:8000 .2. SHARE — Broadcast to the LAN Open a LAN compute pool with shard lan create --open (or generate a pairing code via shard lan create ). SHARD broadcasts UDP discovery beacons on port 8001 and exposes /api/lan/* endpoints so peer machines on your subnet can pool their VRAM and RAM.
3. CONNECT — Pair Worker Nodes On any secondary computer on the network, run shard lan join <master-ip> --share . The worker completes an HTTP handshake ( POST /api/lan/pair ), launches its local RPC tensor worker, sends periodic health heartbeats, and contributes its GPU/RAM to the cluster.4. USE — Power AI Agents & Apps Connect coding IDEs, autonomous AI agents, or custom Python apps to http://<master-ip>:8000/v1/chat/completions or /v1/messages . Master coordinates layer execution across connected nodes and streams tokens back via Server-Sent Events (SSE).

1. CREATE — Launch Your Server

Step 1: Create and Launch Your SHARD Server
Step 1: Create and Launch Your SHARD Server

Start a local SHARD node with shard start or python main.py. The engine probes your CPU, system RAM, and NVIDIA CUDA / Apple MPS GPU via hardware.py, loads your chosen GGUF or Hugging Face model, and brings up the FastAPI server and Web UI on http://localhost:8000.

2. SHARE — Broadcast to the LAN

Step 2: Share Server With the LAN Network
Step 2: Share Server With the LAN Network

Open a LAN compute pool with shard lan create --open (or generate a pairing code via shard lan create). SHARD broadcasts UDP discovery beacons on port 8001 and exposes /api/lan/* endpoints so peer machines on your subnet can pool their VRAM and RAM.

3. CONNECT — Pair Worker Nodes

Step 3: Connect and Pair to a SHARD Server
Step 3: Connect and Pair to a SHARD Server

On any secondary computer on the network, run shard lan join <master-ip> --share. The worker completes an HTTP handshake (POST /api/lan/pair), launches its local RPC tensor worker, sends periodic health heartbeats, and contributes its GPU/RAM to the cluster.

4. USE — Power AI Agents & Apps

Step 4: Host and Access AI Agents Through SHARD
Step 4: Host and Access AI Agents Through SHARD

Connect coding IDEs, autonomous AI agents, or custom Python apps to http://<master-ip>:8000/v1/chat/completions or /v1/messages. Master coordinates layer execution across connected nodes and streams tokens back via Server-Sent Events (SSE).

Features

Every feature listed below is verified directly against the SHARD source tree (v1.8.1).

Implemented Features

CategoryCapabilitySource ModuleDescription
LAN Cluster & ShardingUDP Auto-Discovery & LAN Poolinglan.py , docs/LAN_SHARDING_DESIGN.mdBroadcasts UDP beacons ( :8001 ), pairs nodes via code or --open mode, monitors node heartbeats ( /api/lan/heartbeat ), and pools VRAM/RAM across machines.
Inference BackendsDual PyTorch & GGUF ( llama.cpp ) Engineengine.py , gguf_backend.pyRuns Hugging Face transformers / safetensors models on CUDA/CPU/MPS and quantized GGUF weights via managed llama-server & ggml-rpc-server subprocesses with configurable GPU layer offload ( -ngl ).
Hardware PlanningPre-Flight Model Fit Advisor ( shard fit )fit.py , hardware.pyInspects model architecture parameters ( config.json or parameter heuristics) against host VRAM and RAM before downloading, returning GPU , HYBRID , or IMPOSSIBLE verdicts, max context length, estimated tok/s, and quantization advice.
Native AccelerationC++20 & MASM AVX2 SIMD Kernelsnative/shard_core.cpp , native/shard_asm.asm , shard_bridge.pyCustom C++20 and x86-64 AVX2 assembly routines for RMSNorm , LayerNorm , Rotary Position Embeddings ( RoPE ), Softmax , GELU , SiLU , and MatMul loaded via ctypes .
API & StreamingOpenAI & Anthropic Compatible Serverserver.pyServes /v1/chat/completions , /v1/completions , /v1/models , /v1/messages (Anthropic protocol), /api/generate , and /api/generate/stream with real-time Server-Sent Events (SSE) and optional API key auth ( shard key ).
Web Control PanelZero-Dependency Dashboard & Playgrounddashboard.htmlBuilt-in browser UI at http://localhost:8000 with live GPU/VRAM/RAM telemetry, Hugging Face model search & download, quantization selector, streaming chat playground, LAN node manager, and one-click IDE integration snippets.
ContainerizationDocker & NVIDIA GPU Compose StackDockerfile , docker-compose.ymlProduction-ready container setup with persistent named volumes ( shard_models_volume , shard_data_volume ) and NVIDIA GPU reservation.
Collaborative TrainingOptional SOUP Fine-Tuning Ecosystemsoup_adapter.py , feedback.py , training.py , docs/COLLABORATIVE_TRAINING.mdPrivacy-first consent feedback, secret/PII redaction, immutable SFT/DPO dataset snapshots, dynamic SOUP training orchestration with cancellation, Baseline vs Candidate eval, model versioning, deployment, and 1-click rollback.
Unified Hosting & PrivacyModel-First Hosting & Private Sessionsserver.py , lan.pyEnforces loaded model verification before server creation/publishing (HTTP 409 MODEL_NOT_READY ), guarantees zero-leakage participant-isolated private chats, and enforces compute worker VRAM/CPU quotas.
Global Network & DiscoveryAutomatic Network & 1-Click Clustersnet_strategy.py , net_discovery.py , cluster.pyAutomatic network initialization on shard start , public Server Catalog with search and VRAM filters, STUN/NAT traversal, blind encrypted WebSocket relay, pre-join hardware probing, explicit consent enforcement, and distributed tensor pipelines.

Experimental Features

  • Multi-Host llama.cpp RPC Tensor Distribution (gguf_backend.py, lan.py): Distributed layer execution over TCP/RPC across heterogeneous LAN GPUs works best on low-latency wired Gigabit/2.5GbE Ethernet subnets; Wi-Fi links may experience higher inter-token latency.
  • Native Win32 Desktop GUI (native/shard_gui.cpp): Native C++ Windows desktop wrapper alongside the cross-platform web dashboard (dashboard.html).

Planned Features

  • Automated Dynamic Re-Sharding on Node Dropout: Seamless mid-generation layer migration if a worker node disconnects unexpectedly (docs/LAN_SHARDING_DESIGN.md).
  • Linux / macOS ARM64 NEON Intrinsics: Expanding the native SIMD kernel suite (native/) beyond x86-64 AVX2/FMA to native Apple Silicon / ARM64 assembly paths.

Installation & Quick Start

Prerequisites

  • OS: Windows 10/11 (64-bit), Linux (Ubuntu 20.04+ / Debian 12+), or macOS.
  • Python: Python 3.8+ (3.10+ recommended).
  • Hardware: NVIDIA GPU with CUDA drivers (recommended for GPU acceleration), Apple Silicon (MPS), or x86-64 CPU with AVX2 support.
  • Network (Optional for LAN Sharding): Two or more machines on the same LAN with TCP port 8000 (HTTP/API) and UDP/TCP port 8001 (LAN discovery & RPC) open.

Installation

Option 1: Install via PyPI (Recommended)

Install SHARD directly from PyPI in any environment:


        # Core install: CLI, FastAPI server, web dashboard, GGUF (llama.cpp) + LAN clustering
pip install shard-engine

# Full install: adds PyTorch/Transformers safetensors & Hugging Face extras
pip install "shard-engine[full]"
      

To install the latest development snapshot directly from GitHub:


        # Core install from GitHub
pip install "git+https://github.com/avelone123123/shard.git"

# Full install from GitHub
pip install "shard-engine[full] @ git+https://github.com/avelone123123/shard.git"
      

Verify your environment right after installation:


        shard doctor
      
Windows PATH Note: If Windows reports that shard.exe is not on your PATH, you can run SHARD directly through Python: python -m shard_engine <command> (for example, python -m shard_engine help or python -m shard_engine start). GPU & Runtime Note: GGUF inference auto-downloads platform binaries for llama-server / ggml-rpc-server on first run and needs no extra ML packages. For CUDA PyTorch inference, install a matching CUDA build of torch before pip install "shard-engine[full]".

Option 2: Editable Install from Source (Developers)


        git clone https://github.com/avelone123123/shard.git
cd shard
pip install -e ".[dev]"
      

Option 3: Docker Compose (with NVIDIA GPU Support)


        git clone https://github.com/avelone123123/shard.git
cd shard
docker-compose up --build
      

This builds shard-engine, mounts persistent volumes (/data/models and /data/shard), exposes port 8000:8000, and enables NVIDIA GPU passthrough.

Option 4: Compile Native C++20 / MASM AVX2 Kernels (Optional Speedup)

To compile shard_core.dll (Windows MSVC cl.exe + ml64.exe) or shard_core.so (Linux GCC/Clang):


        python native/build_native.py
      

Upgrading & Uninstalling


        shard update --check
pip install --upgrade shard-engine
pip uninstall shard-engine
      

Uninstalling the Python package keeps your downloaded models, settings, and API keys intact in your user data directory.

Quick Start Workflow

1. Verify Environment & Launch SHARD


        # Check Python runtime, dependencies, GPU VRAM, ports, and writable paths
shard doctor

# Start FastAPI server in background (--daemon / -d) and inspect logs/status
shard start --daemon
shard status

# Or open the interactive Web Dashboard at http://localhost:8000 immediately
shard gui

# Stop or restart background server cleanly
shard restart
shard stop
      

2. Check Hardware & Model Fit Before Downloading


        # Check if a model fits your local machine or LAN cluster before downloading
shard fit Qwen/Qwen2.5-Coder-7B-Instruct
shard fit meta-llama/Llama-3.1-8B-Instruct
      

3. Download & Run a Model


        # Download and start an interactive terminal chat (default: Qwen/Qwen2.5-Coder-1.5B-Instruct)
shard run Qwen/Qwen2.5-Coder-7B-Instruct

# List cached and catalog models
shard list
      

4. Create or Join a Distributed LAN Cluster

On the Master Machine (shard lan create prints the live LAN IPv4 addresses detected on this host):


        # Create an open LAN pool (auto-accepts subnet workers and sets target model)
shard lan create --open --share --model Qwen/Qwen2.5-Coder-7B-Instruct

# Or create a code-protected LAN pool
shard lan create --share
      

On each Worker Machine on the same subnet, replace <SERVER_IP> with the IP printed by shard lan create on the Master Machine:


        # Join an open LAN pool and contribute local GPU/RAM
shard lan join <SERVER_IP> --share

# Or join a code-protected pool
shard lan join <SERVER_IP> <PAIR_CODE> --share

# Inspect cluster nodes, host IPv4 addresses, and pooled VRAM/RAM
shard lan status

# Leave or disable LAN pooling
shard lan off
      

CLI Reference & Configuration

Verified CLI Commands (shard_engine/cli.py)

CommandArguments & FlagsDescription
shard start[--host HOST] [--port PORT] [-d│--daemon] [--no-browser]Launches the SHARD FastAPI server and Uvicorn runtime (default 0.0.0.0:8000 ).
shard stop(none)Safely stops the background SHARD server managed by PID tracking.
shard restart[--host HOST] [--port PORT] [--foreground] [--no-browser]Stops and restarts the SHARD server cleanly.
shard gui(none)Launches the server (or native Win32 desktop UI if present) and opens the Web Dashboard.
shard status(none)Displays detected CPU, RAM, GPU VRAM, active model, and detected LAN IPv4 addresses.
shard doctor(none)Audits Python, packaged assets, dependencies, CUDA/GPU, LAN IPs, ports, and user directories.
shard version[-V│--version]Prints installed SHARD version, Python executable path, OS info, and active directories.
shard update[--check]Checks PyPI for newer shard-engine releases or upgrades non-destructively via pip .
shard fit<model_id> [--ctx N] [--no-fetch]Calculates VRAM/RAM requirements, quantization fit, and estimated throughput before downloading.
shard run[model_id]Downloads (if needed), loads the model, and opens interactive terminal chat REPL.
shard list[--category CAT]Lists cached and curated catalog models ( all , ideal , coding , reasoning , compact ).
shard lan[status│create│join│model│off] [extra] [--share] [--open] [--model ID] [--port N]Creates, joins, configures, inspects, or stops distributed LAN compute pooling across machines.
shard network[status│doctor│radmin [status│install│repair│gui│decline]]Inspects multi-tier autonomous network, runs network doctor, and manages automated Radmin VPN fallback.
shard training[status│doctor│privacy│feedback│dataset│start│logs│cancel│resume│models│eval│prepare│approve│deploy│rollback]Manages consent feedback, dataset snapshots, SOUP fine-tuning jobs, evaluation, model deployment, and rollback ( Guide ).
shard key[--gen]Views the current local API key or generates a new token ( --gen ).
shard models-dir[path │ reset]Views, updates, or resets the directory where model weights are stored.
shard test[--strict-native]Verifies tensor math kernels and AVX2/pure-Python fallback throughput.
shard help[-h│--help]Prints full CLI command usage and examples.

Cross-Platform Directories & Environment Variables (shard_engine/paths.py)

SHARD stores user data outside site-packages following OS conventions (and reads existing ~/.shard data cleanly on upgrade):

  • Windows: %LOCALAPPDATA%\SHARD (config, keys_db.json, logs/, run/, models/)
  • Linux: ~/.local/share/shard, ~/.config/shard, ~/.cache/shard, ~/.local/state/shard (respects XDG_* variables)
  • macOS: ~/Library/Application Support/SHARD, ~/Library/Caches/SHARD, ~/Library/Logs/SHARD
Variable / FileDefaultPurpose
SHARD_HOST0.0.0.0Network interface IP address bound by server.py .
SHARD_PORT8000HTTP port for the API server and Web Dashboard.
SHARD_DATA_DIROS data directory ( /data/shard in Docker)Overrides root directory for settings.json , keys_db.json , lan.json , logs, and PID metadata.
SHARD_MODELS_DIR<data_dir>/models ( /data/models in Docker)Storage directory for downloaded Hugging Face SafeTensors and GGUF weights.
SHARD_NO_BROWSER / SHARD_DOCKER(unset)Suppresses auto-opening a browser tab when starting SHARD.
SHARD_NATIVE_LIB(auto-discovered)Path override to shard_core.dll / .so / .dylib (set none to force Python fallback).
SHARD_SOUP_PYTHON / SHARD_SOUP_CMD(auto-discovered)Optional path to a Python 3.11–3.13 environment or binary running soup-cli when training on Python 3.14+ hosts.
HF_HOME / TRANSFORMERS_CACHE~/.cache/huggingfaceStandard Hugging Face cache override respected by engine.py and hf_hub.py .
engine_optimizations.jsonPackaged inside shard_engineRuntime optimization profile loaded via importlib.resources .

Architecture

SHARD Ecosystem Topology: Coding IDEs, agent frameworks, master coordinator, and distributed LAN worker nodes operating as a unified inference fabric.

SHARD Network Ecosystem Overview
SHARD Network Ecosystem Overview

System Architecture Diagram


        flowchart TB
    subgraph Clients["AI Agents, IDEs & SDK Clients"]
        IDE["Cursor / Claude Code / Aider / Continue / Zed"]
        SDK["OpenAI SDK / Anthropic SDK / LangChain / LlamaIndex"]
        Browser["Web Dashboard UI (dashboard.html)"]
    end

    subgraph Master["SHARD Master Node (FastAPI :8000)"]
        Server["server.py\n(/v1/chat/completions, /v1/messages, /api/*)"]
        Fit["fit.py & hardware.py\n(Pre-Flight VRAM/RAM Advisor)"]
        Engine["engine.py\n(PyTorch Transformers & Token Streamer)"]
        GGUF["gguf_backend.py\n(llama-server & Layer Offload -ngl)"]
        Bridge["shard_bridge.py\n(ctypes C++20 / MASM AVX2 SIMD Core)"]
        LANCoord["lan.py\n(UDP Discovery :8001 & Cluster Coordinator)"]
    end

    subgraph Workers["Distributed LAN Worker Nodes (:8001)"]
        W1["Worker Node 1\nggml-rpc-server / GPU VRAM Shard"]
        W2["Worker Node 2\nggml-rpc-server / GPU VRAM Shard"]
        W3["Worker Node N\nggml-rpc-server / System RAM + GPU"]
    end

    IDE -->|HTTP / SSE :8000| Server
    SDK -->|HTTP / SSE :8000| Server
    Browser -->|REST / SSE :8000| Server

    Server --> Fit
    Server --> Engine
    Server --> LANCoord
    Engine --> GGUF
    Engine --> Bridge

    LANCoord <-->|UDP Broadcast & Heartbeat :8001| W1
    LANCoord <-->|UDP Broadcast & Heartbeat :8001| W2
    LANCoord <-->|UDP Broadcast & Heartbeat :8001| W3

    GGUF <-->|TCP Tensor RPC Stream| W1
    GGUF <-->|TCP Tensor RPC Stream| W2
    GGUF <-->|TCP Tensor RPC Stream| W3
      

Core Repository Modules


        shard/
├── assets/
│   ├── branding/              # Official SHARD SVG & PNG logos, icons, and favicon
│   ├── animations/            # Isometric pixel-art / voxel animated GIFs
│   └── screenshots/           # Dashboard and CLI fit advisor visuals
├── docs/
│   └── LAN_SHARDING_DESIGN.md # Distributed LAN sharding protocol & architecture spec
├── native/
│   ├── build_native.py        # Cross-platform compiler for C++20 & MASM AVX2 kernels
│   ├── shard_core.cpp         # C++20 RMSNorm, LayerNorm, RoPE, Softmax, GELU, SiLU, MatMul
│   ├── shard_asm.asm          # x86-64 MASM AVX2 SIMD assembly kernels
│   └── shard_gui.cpp          # Native Win32 desktop GUI wrapper
├── shard_engine/                # Installable Python package (paths, process, CLI, API, inference)
│   ├── paths.py                 # Cross-platform XDG / AppData / macOS paths & packaged assets
│   ├── process.py               # Detached daemon launcher & verified PID lifecycle management
│   ├── cli.py                   # `shard` command-line interface entry point (`shard_engine.cli:main`)
│   ├── main.py                  # Application launcher & Uvicorn bootstrapper
│   ├── server.py                # FastAPI server (OpenAI, Anthropic, Model & LAN routes)
│   ├── engine.py                # Core LLM inference orchestrator & SSE generator
│   ├── gguf_backend.py          # Managed llama.cpp / GGUF server & RPC integration
│   ├── lan.py                   # Dynamic subnet IP discovery, UDP beacons & LAN pooling
│   ├── fit.py                   # Pre-flight model memory & throughput calculator
│   ├── hardware.py              # GPU VRAM (nvidia-smi), CPU & RAM probing
│   ├── hf_hub.py                # Hugging Face Hub search, metadata & downloader
│   ├── shard_bridge.py          # Python ctypes bridge to native shard_core.dll / .so
│   ├── settings.py              # Persistent configuration manager
│   └── dashboard.html           # Packaged web control panel & agent config generator
├── pyproject.toml               # Single-source packaging configuration (version from __init__.py)
├── tests_shard.py               # Unit, API, fit advisor, LAN sharding & kernel verification suite
└── tests_cli_pkg.py             # Packaging, OS path, CLI, PID safety & 2-node LAN cluster suite
      

Web Dashboard & Demonstrations

Interactive Web Control Panel (dashboard.html)

Navigating to http://localhost:8000 opens the built-in SHARD Control Panel:

SHARD Web Dashboard: Real-time per-node VRAM/RAM gauges, Pre-Flight Model Fit Advisor, one-click IDE/agent config snippets, and streaming playground.

SHARD Web Dashboard & LAN Pool Telemetry
SHARD Web Dashboard & LAN Pool Telemetry

CLI Pre-Flight Advisor (shard fit) & LAN Cluster Status (shard lan status)

Pre-Flight Fit Verification: Evaluate model quantization, context ceiling, and pooled LAN cluster fit before downloading model weights.

SHARD CLI Fit Advisor and LAN Status
SHARD CLI Fit Advisor and LAN Status

Connecting AI Agents & Coding Tools

Point any OpenAI- or Anthropic-compatible client to your SHARD instance:


        from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="shard-local",  # Or generate a token via `shard key --gen`
)

stream = client.chat.completions.create(
    model="Qwen/Qwen2.5-Coder-7B-Instruct",
    messages=[{"role": "user", "content": "Explain tensor parallelism across LAN nodes."}],
    stream=True,
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
      

Distributed AI Cluster Mode

SHARD supports collaborative model-parallel execution across independent machines (LAN, Radmin VPN, Tailscale, WireGuard, or NAT-traversed encrypted tunnels). A coordinator can schedule a large model across multiple consented worker nodes and expose one unified OpenAI/Anthropic endpoint (/v1/chat/completions, /v1/messages).

Quick Start (Coordinator + Workers)

  • Coordinator (Create & Publish Global Server): shard start --daemon shard cluster create Qwen/Qwen2.5-Coder-7B-Instruct --name MyGlobalCluster --strategy auto --publish --slots 4 shard cluster status

Coordinator (Create & Publish Global Server):


        shard start --daemon
shard cluster create Qwen/Qwen2.5-Coder-7B-Instruct --name MyGlobalCluster --strategy auto --publish --slots 4
shard cluster status
      
  • Discover Global Servers & 1-Click Join: # Browse public catalog shard cluster catalog --q deepseek --min-vram 16 # Join as an AI Consumer (Client): shard cluster join shn_coordinator_node_id --client # Join as a Compute Worker (Explicit Consent & Quota): shard cluster join shn_coordinator_node_id --consent --vram-pct 70 --max-ram-gb 8 --max-threads 4 For workers behind restrictive CGNAT/NAT, SHARD automatically traverses NAT using STUN or establishes an end-to-end ChaCha20-Poly1305 encrypted blind WebSocket relay tunnel with zero configuration.

Discover Global Servers & 1-Click Join:


        # Browse public catalog
shard cluster catalog --q deepseek --min-vram 16

# Join as an AI Consumer (Client):
shard cluster join shn_coordinator_node_id --client

# Join as a Compute Worker (Explicit Consent & Quota):
shard cluster join shn_coordinator_node_id --consent --vram-pct 70 --max-ram-gb 8 --max-threads 4
      

For workers behind restrictive CGNAT/NAT, SHARD automatically traverses NAT using STUN or establishes an end-to-end ChaCha20-Poly1305 encrypted blind WebSocket relay tunnel with zero configuration.

  • Deploy & Monitor: shard cluster plan shard cluster deploy shard cluster status

Deploy & Monitor:


        shard cluster plan
shard cluster deploy
shard cluster status
      

Supported Execution Backends & Documentation

  • GGUF Distributed RPC (gguf_rpc_layer_split): Splits model layers and KV cache across ggml-rpc-server workers and local offload devices via llama-server --rpc, with continuous batching (--parallel, --cont-batching).
  • PyTorch Pipeline Parallelism (torch_pipeline_parallel): Directly loads only assigned safetensors layer shards per worker (llama, qwen2, qwen3, mistral, gemma, gemma2, phi3, gpt2), maintains per-worker distributed KV caches, and streams binary tensor activations over authenticated SHPT TCP sockets.
  • Autonomous Global Network & Infrastructure Specs: Autonomous Architecture: AUTONOMOUS_NETWORK_ARCHITECTURE.md Radmin VPN Integration & Fallback: RADMIN_VPN_INTEGRATION_REPORT.md Production Deployment & Docker: SHARD_PRODUCTION_INFRASTRUCTURE_REPORT.md Autonomous Installation & OS Workflows: AUTONOMOUS_INSTALLATION_REPORT.md Real-World Test Verification Report: REAL_WORLD_NETWORK_TEST_REPORT.md Catalog & Discovery: GLOBAL_SERVER_DISCOVERY_REPORT.md 1-Click Worker Onboarding: AUTOMATIC_WORKER_JOIN_REPORT.md
  • Autonomous Architecture: AUTONOMOUS_NETWORK_ARCHITECTURE.md
  • Radmin VPN Integration & Fallback: RADMIN_VPN_INTEGRATION_REPORT.md
  • Production Deployment & Docker: SHARD_PRODUCTION_INFRASTRUCTURE_REPORT.md
  • Autonomous Installation & OS Workflows: AUTONOMOUS_INSTALLATION_REPORT.md
  • Real-World Test Verification Report: REAL_WORLD_NETWORK_TEST_REPORT.md
  • Catalog & Discovery: GLOBAL_SERVER_DISCOVERY_REPORT.md
  • 1-Click Worker Onboarding: AUTOMATIC_WORKER_JOIN_REPORT.md
  • Technical Specs: DISTRIBUTED_AI_ARCHITECTURE.docx | DISTRIBUTED_AI_IMPLEMENTATION_REPORT.docx | DISTRIBUTED_AI_TEST_REPORT.docx

Project Status & Community

Technology Stack

  • Runtime & API: Python 3.8+ / 3.10+, FastAPI, Uvicorn, Pydantic v2
  • ML & Quantization: PyTorch, Hugging Face transformers, accelerate, safetensors, huggingface_hub, llama.cpp GGUF
  • Native Acceleration: C++20 (native/shard_core.cpp) + x86-64 MASM AVX2 Assembly (native/shard_asm.asm)
  • System & Networking: psutil, UDP broadcast discovery (:8001), HTTP/SSE streaming (:8000)

Community, Contribution & Governance

  • Architecture & LAN Design Spec: docs/LAN_SHARDING_DESIGN.md
  • Contribution Guidelines: CONTRIBUTING.md — covers zero-copy memory rules, native C++20/AVX2 build steps, and running shard test.
  • Code of Conduct: CODE_OF_CONDUCT.md
  • Release History: CHANGELOG.md (current release: v1.8.1)
  • Bug Reports & Feature Requests: GitHub Issues
  • Community Q&A & Ideas: GitHub Discussions
  • Security Policy: Please report security vulnerabilities privately via GitHub Security Advisories or by contacting the maintainer before public disclosure.
  • License: Distributed under the MIT License.

SHARD — One Network. Many Servers. Shared AI Infrastructure. Built for the open-source local AI community · English · Русский · Қазақша

The text above was extracted automatically from the source page and may miss images, tables or code blocks. The original is authoritative.

Published by HN AI
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗