Pal-E World

Project: Pal-E World

Vision

A Palworld cloud gaming platform on k8s. Rent on-demand Vast.ai GPUs, run GOW (Games on Whales) containers with Sunshine streaming, connect via Moonlight from any device. No local gaming GPU required — archbox is a thin client, Hyprland stays running.

Target audience: Arch Linux + Hyprland users (dogfooding our own setup). Palworld 1.0 launches July 10, 2026.

Status

• Dedicated server: IaC in palworld-server repo. REST API enabled (port 8212), 16Gi RAM, RCON retained with deprecation note.
• Helm deploy + headless GPU: Merged (palworld-server PR #9). Fixed Helm boolean/port-array bugs, added ConnectedMonitor xorg config for NVENC without host compositor. Docs restructured with mermaid diagrams and user stories.
• Sunshine self-hosted: Merged (palworld-server PR #14). Harbor-hosted Sunshine image (harbor.tail5b443a.ts.net/pal-e/sunshine:latest), parameterized all container images, LOG_LEVEL env var, init container fixes state file permissions. GoW source mirrored to Forgejo (ldraney/sunshine).
• Tailscale streaming fixes: Merged (palworld-server PR #15). Injected origin_pin_allowed = wan for Tailscale pairing, increased ping_timeout to 10000ms to prevent VPN jitter disconnects. Documented curl-based PIN workflow with 30s timing window.
• iOS Moonlight streaming: Confirmed working (2026-07-11). HEVC encoding, Xbox One gamepad detected and stable. iPhone must be on WiFi — cellular + Tailscale causes RTSP ETIMEDOUT (error 60). Controller bug fixed: default ping_timeout (2000ms) was too aggressive for Tailscale.
• Local docker-compose streaming: Merged (palworld-server PR #19). Full GoW stack in docker/ for local streaming. Modesetting xorg driver + KMS capture + libx264 encoding. Sunshine config override mechanism for Tailscale WAN pairing. Terraform renamed to tofu/. Docs split into docs/dev/ and docs/prod/. Vulkan black screen fixed with windowed launch option (#20 closed). MacBook streaming over Tailscale confirmed working.
• NvFBC/NVENC blocked: Issue #22 open. NvFBC capture fails with OpenGL 0x505 in containers — mesa DRI3 can't create GL context for nvidia-drm. Both docker and k8s deployments use libx264 software encoding (50-80ms/frame lag). GPU encoding is the #1 performance blocker.
• Vast.ai integration: Restored (pal-e-platform PR #515, reverts #507). External data source queries GPU offers. RTX 4090 ~$0.14/hr.
• Salt client states: Merged (PR #478). gaming-session start/teardown scripts, SOPS-encrypted API key, gaming pillar.
• Vast.ai bootstrap: Merged (PR #481). Docker Compose GOW stack, onstart.sh (Tailscale + docker compose up), start.sh with MagicDNS resolution.
• Tailscale auth key: Merged (PR #484). Terraform-managed reusable+ephemeral key, tag:gaming ACL. No manual key generation needed.
• Client-side docs: Merged (PR #12, hyprland-config). README, architecture, pal-e-world.md updated.
• E2E smoke test tooling: Merged (PR #482). Preflight check script, smoke test results template. Manual test pending.

User Stories

Story	Description	Status
pal-e-world	As a Palworld player, I want to launch a cloud gaming session from my Arch workstation so I can play without local GPU conflicts	active

Architecture

archbox (thin client)                    Vast.ai (GPU rental)
├── Hyprland (stays running)             ├── RTX 4090 (~$0.14/hr)
├── Moonlight ────── streams from ────────►├── GOW Pod
├── Ollama (local GPU for LLM)           │   ├── xorg (modesetting :99)
└── k3s (palworld-server, CPU-only)      │   ├── pulseaudio (virtual audio)
                                         │   ├── sunshine (KMS + libx264)
Local mode (docker/):                    │   └── steam (Proton + Palworld)
├── xorg (modesetting :0, host network)  └── Tailscale (private network)
├── sunshine (KMS capture → libx264)
├── steam (game client)                  DNS: GoDaddy → Hetzner edge VPS (Caddy + Tailscale) → k3s
└── moonlight (linuxfb → /dev/fb0)       IaC: OpenTofu (Vast.ai external data source) + Salt (client config)

Roadmap

Phase	Status	Summary
Phase 1: Revival	COMPLETE	Unarchive repo (#468 ✓), Vast.ai provider (#474 ✓), REST API (#470 ✓), Salt client (#469 ✓)
Phase 2: Platform	IN PROGRESS	Docs (#1628 ✓), bootstrap (#479 ✓), smoke test tooling (#480 ✓), Tailscale key (#483 ✓), Harbor Sunshine (#14 ✓), Tailscale pairing (#15 ✓), iOS streaming ✓, controller ✓, local docker streaming (#19 ✓), modesetting/KMS capture (#21 ✓), Vulkan fix (#20 ✓). Remaining: NvFBC/NVENC (#22), manual E2E test, landing page (#471 parked)
Phase 3: Scale	planned	Server clustering (#472), multi-user streaming, NVENC encoding (depends on #22)

Repos

Repo	Platform	Role	Status
ldraney/palworld-server	Forgejo	GOW pod + dedicated server IaC	active
ldraney/sunshine	Forgejo	GoW Sunshine source mirror (Harbor CI builds)	active
ldraney/pal-e-platform	Forgejo	Salt states, Vast.ai Terraform, platform infra	active
ldraney/hyprland-config	Forgejo	Client-side Hyprland + Moonlight setup	active
ldraney/godaddy-tofu	Forgejo	Custom OpenTofu provider for GoDaddy DNS	active

Key Dates

• 2026-07-10: Palworld 1.0 — server clustering, RCON deprecated → REST API (port 8212), 16GB RAM minimum
• 2026-07-11: iOS Moonlight streaming confirmed working (PR #14 merged). Tailscale pairing + controller stability fixed (PR #15 merged).
• 2026-07-17: Local docker-compose streaming merged (PR #19). Modesetting/KMS/libx264 pipeline working. MacBook streaming over Tailscale confirmed. NvFBC bug filed (#22).

Resume Notes

• Phase 1 complete — all 4 tickets merged
• Phase 2: all IaC merged (12 PRs total). Local docker streaming working with modesetting/KMS/libx264. MacBook + iOS streaming verified. NvFBC (#22) is the #1 perf blocker — libx264 adds 50-80ms encode latency.
• Landing page (#471) parked — needs domain name from user + decomposition
• $24.82 Vast.ai credit available (~177 hours of RTX 4090)

Verdict: APPROVED

Re-review after scope refinement. All three [BODY] issues from review-1650-2026-06-28 have been fixed. The issue body is now complete, file targets are verified, and acceptance criteria use the correct Terraform-based approach. Two [SCOPE] items remain as non-blocking recommendations.

Template Completeness

• [x] Type -- Feature
• [x] Lineage -- Standalone, part of Phase 3 pivot
• [x] Repo -- ldraney/pal-e-platform
• [x] User Story -- well-formed
• [x] Context -- includes Terraform rationale (kubectl would be reverted by tofu apply)
• [x] File Targets -- specific: terraform/modules/ops/main.tf (helm_release.ollama). Fixed from previous review.
• [x] Feature Flag -- "none"
• [x] Acceptance Criteria -- 5 items, all Terraform-based. Fixed from previous review.
• [x] Test Expectations -- 3 items
• [x] Constraints -- 3 items (reversibility, preserve PVCs, no side effects)
• [x] Checklist -- now includes "Tests pass". Fixed from previous review.
• [x] Related -- project and downstream issue referenced

Traceability

• [x] story:gpu-freedom label -- GPU Freedom story
• [x] story note verified -- story-pal-e-world-gpu-freedom exists in pal-e-docs (user-story type, active status)
• [ ] story note on project page -- NOT listed in project-pal-e-world user-stories table. [SCOPE] Add gpu-freedom entry to project-pal-e-world user-stories section.
• [x] arch:k3s label -- k3s infrastructure
• [ ] arch note MISSING -- no arch-k3s note found in pal-e-docs. [SCOPE] Create architecture note arch-k3s for k3s component, or relabel to arch:ollama (consistent with review-1552-2026-06-22-r2).
• [x] Forgejo issue -- ldraney/pal-e-platform#486, open

File Targets

• [x] terraform/modules/ops/main.tf -- VERIFIED. File exists in pal-e-platform repo. helm_release.ollama resource found at line 47. Contains GPU config (nvidia runtime, gpu.enabled=true, gpu.number=1), model pull (qwen3-embedding:4b), hostPath volume, and depends_on nvidia_device_plugin. Exact match to issue description.
• [x] Exclusions correctly scoped -- pal-e-docs code (separate Groq migration issue pal-e-api #283) and Salt states (host Ollama already disabled).

Repo Placement

OK. Issue filed on ldraney/pal-e-platform, file target is in pal-e-platform. Single repo -- no cross-repo issues needed.

Dependencies

• Board item #1653 ("Groq API migration for pal-e-docs embeddings", story:gpu-freedom, 5 pts) depends on THIS ticket completing first. Documented in issue Context and Related sections. Sequencing correct.
• No items in in_progress that block this work.
• No undocumented blockers detected.

Acceptance Criteria

All 5 AC are now Terraform-based and verifiable (fixed from previous review):

• [x] AC1: "Terraform Helm values set Ollama replicas to 0 (or release disabled via count conditional)" -- verifiable by code review of helm_release.ollama
• [x] AC2: "tofu plan shows the expected change (scale down, not destroy)" -- verifiable by running tofu plan
• [x] AC3: "After tofu apply, nvidia-smi shows no Ollama processes" -- verifiable manually
• [x] AC4: "pal-e-docs semantic search degrades to keyword-only without crashing" -- verifiable manually
• [x] AC5: "Change is reversible -- can set replicas back to 1 for re-embedding" -- verifiable by code review

Blast Radius

• pal-e-docs semantic search -- acknowledged in issue. Degrades to keyword-only. Covered by AC4.
• Embedding worker (pal-e-api/k8s/embedding-worker.yaml) -- runs with replicas=1, connects to http://ollama.ollama.svc.cluster.local:11434. Will get ConnectError when Ollama scales down. Worker has error handling (marks blocks as errored, continues polling). Will generate embedding_errors_total Prometheus metrics. Acceptable temporary state before Groq migration (#1653). Not a blocker.
• NVIDIA device plugin -- separate Helm release in same module, not affected.
• Salt states -- ollama-service: service.dead already in place for host systemd. No conflict.
• Other ops module resources -- terraform/modules/ops/main.tf also contains TF state backup CronJob and NVIDIA device plugin. Change is scoped to helm_release.ollama only. Constraints section correctly warns "Must not affect other resources."

Decomposition Assessment

1 file target, 1 repo, 5 acceptance criteria (3 are manual verification, not code changes). Estimated agent work well under 5 minutes. No decomposition needed.

Recommendation

All [BODY] issues from the previous review are resolved. The issue spec is now complete and implementable. Two [SCOPE] items remain as non-blocking recommendations for traceability hygiene:

• [SCOPE] Add gpu-freedom story entry to project-pal-e-world user-stories table (story note exists but is not registered on the project page).
• [SCOPE] Create architecture note arch-k3s for k3s component. Or relabel board item to arch:ollama if preferred (consistent with prior review review-1552-2026-06-22-r2).

These are parallel documentation tasks that do not block ticket implementation.

Verdict: NEEDS_REFINEMENT

Re-review of board item #1652 after scope refinement. All [BODY] issues from the previous review (review-1652-2026-06-28) have been resolved. Only [SCOPE] items remain.

Template Completeness

Template: template-issue-bug

• [x] Type — Bug
• [x] Lineage — Standalone, discovered during Pal-E World Phase 2 testing on 2026-06-28
• [x] Repo — ldraney/hyprland-config
• [x] What Broke — Steam overlay pipe crash every 1-3 minutes on XWayland/Hyprland
• [x] Repro Steps — 4 clear steps provided
• [x] Expected Behavior — Present, specifies -no-overlay fix
• [x] Environment — archbox, GTX 1070, Hyprland + XWayland, Steam stable
• [x] File Targets — 3 targets with line numbers (added since previous review)
• [x] Acceptance Criteria — 4 criteria (increased from 3, added docs update AC)
• [x] Related — pal-e-world project referenced

All required Bug template sections present. File Targets section is an extension beyond the base template — well-structured.

Traceability

• [x] story:remote-play label — present on board item
• [ ] story note MISSING — project-pal-e-world user-stories section lists "pal-e-world" but no "remote-play" entry. [SCOPE] Add "remote-play" user story entry to project-pal-e-world user-stories section. Three board items (#1651, #1652, #1654) use this label.
• [x] arch:infra label — present on board item
• [ ] arch note MISSING — search for arch-infra returned zero results in pal-e-docs. [SCOPE] Create architecture note arch-infra for component infra. Multiple board items reference this label.
• [x] Forgejo issue — ldraney/hyprland-config#14, state: open

File Targets

• [x] hyprland.lua:137 — verified: line 137 contains hl.exec_cmd("pgrep -x steam || steam"). Fix correctly targets adding -no-overlay flag.
• [x] docs/architecture.md:76 — verified: line 76 reads "Known limitation: Steam overlay pipe crash (pipes.cpp:900) every 1-3 minutes on XWayland. Unfixable."
• [x] docs/gpu-sharing-gow.md:5 — verified: line 5 contains "has an unfixable Steam overlay crash (pipes.cpp:900)"
• [ ] docs/gpu-sharing-gow.md:65 — NOT LISTED as file target. Blast radius finding: this line also says "Steam overlay pipe crash every 1-3 minutes on XWayland. Unfixable on Hyprland's XWayland." The AC "Docs updated to reflect the working fix (no longer 'Unfixable')" covers this implicitly, but adding the explicit file target would prevent a stale reference. [BODY] Add docs/gpu-sharing-gow.md:65 as a file target.

Repo Placement

OK — issue filed on ldraney/hyprland-config, all file targets (hyprland.lua, docs/) live in that repo. Single repo, no cross-repo concerns.

Dependencies

• #1651 "Install Sunshine on archbox" (3pt, story:remote-play) — same story, independent. Sunshine streaming is a separate concern.
• #1654 "E2E: MacBook → Moonlight → Tailscale → archbox" (2pt, story:remote-play) — soft dependency; E2E benefits from Steam stability but is not blocked.
• #1650 "Scale down k8s Ollama to free GPU" (1pt, story:gpu-freedom) — independent, different story.
• No items in in_progress that block this ticket. Phase 1 is in_progress but unrelated.
• Dependencies acceptable for standalone bug — not documented in issue body, not needed.

Acceptance Criteria

4 criteria — reasonable count, under the 5 threshold.

• "Steam launches with -no-overlay flag via hyprland.lua" — agent-verifiable (grep for flag in file).
• "Steam runs without crashing for 30+ minutes" — manual testing only. Testable but not automatable.
• "Docs updated to reflect the working fix (no longer 'Unfixable')" — agent-verifiable (grep for "Unfixable" should return zero hits post-fix).
• "No regression in Hyprland session stability" — manual testing only. Testable but not automatable.

AC quality is solid. The docs AC is broad enough to cover all "Unfixable" references including the unlisted line 65.

Blast Radius

• docs/gpu-sharing-gow.md:65 — second "Unfixable" reference in same file, not listed as file target (see File Targets section above).
• docs/pal-e-world.md:73-94 — already documents the crash and -no-overlay solution correctly. No change needed.
• README.md — mentions Steam but makes no "Unfixable" claims. No change needed.
• A worktree .claude/worktrees/steam-crash-fix already exists, suggesting prior implementation work was started. Implementer should check for stale state.

Decomposition Assessment

No decomposition needed.

• File targets: 3 files in 1 repo (plus 1 additional line in same file)
• Acceptance criteria: 4 (under threshold of 5)
• Estimated agent work: under 5 minutes
• Points: 1 (correctly sized)

Recommendations

Resolved from previous review:

• [BODY] Add file targets — RESOLVED: 3 file targets with line numbers now specified
• [BODY] Add AC for "Unfixable" docs — RESOLVED: AC now includes "Docs updated to reflect the working fix"

Remaining:

• [SCOPE] Add "remote-play" user story entry to project-pal-e-world user-stories section. Three board items (#1651, #1652, #1654) use this story label but no backing entry exists.
• [SCOPE] Create architecture note arch-infra for component infra. Multiple board items reference this arch label.
• [BODY] Minor: Add docs/gpu-sharing-gow.md:65 as a fourth file target — second "Unfixable" reference in the same file. The AC covers this implicitly, but explicit targeting prevents oversight.

Verdict: NEEDS_REFINEMENT

Template Completeness

• [x] Type -- Feature
• [x] Lineage
• [x] Repo
• [x] User Story
• [x] Context
• [ ] File Targets -- present but vague, no specific file path
• [x] Feature Flag -- "none"
• [x] Acceptance Criteria -- 3 items
• [x] Test Expectations -- 2 items
• [x] Constraints
• [ ] Checklist -- missing "Tests pass" item from template
• [x] Related

Traceability

• [x] story:gpu-freedom label -- GPU Freedom story
• [ ] story note on project page -- story note story-pal-e-world-gpu-freedom EXISTS in pal-e-docs but is NOT listed in the project-pal-e-world user-stories table. [SCOPE] Add gpu-freedom entry to project-pal-e-world user-stories section.
• [x] arch:k3s label -- k3s infrastructure
• [ ] arch note MISSING -- no arch-k3s note found in pal-e-docs. [SCOPE] Create architecture note arch-k3s for k3s component. Alternatively, consider whether arch:ollama is a more accurate label (prior review review-1552-2026-06-22-r2 used arch:ollama for similar Ollama work).
• [x] Forgejo issue -- ldraney/pal-e-platform#486, open

File Targets

• [ ] "K8s Ollama deployment manifest" -- ISSUE: Not a specific file path. The actual Ollama deployment is a Terraform Helm release at terraform/modules/ops/main.tf (resource helm_release.ollama). The file target must specify this path.
• [x] "pal-e-docs code -- separate Groq migration issue" -- correctly scoped out

Repo Placement

OK. The Ollama Helm release is managed in ldraney/pal-e-platform at terraform/modules/ops/main.tf. Repo assignment is correct. Single repo -- no cross-repo issues needed.

Dependencies

• Board item #1653 ("Groq API migration for pal-e-docs embeddings", story:gpu-freedom) depends on THIS ticket completing first. The issue Context section acknowledges this: "Groq API replaces Ollama for LLM inference." Sequencing is correct.
• No items currently in in_progress that block this work.
• Dependencies are implicitly documented in Context but not explicitly listed.

Acceptance Criteria

ISSUE: The AC reference kubectl scale deployment -n ollama --replicas=0 as the mechanism. However, the Ollama deployment is Terraform-managed via helm_release.ollama. A manual kubectl scale would be reverted on the next tofu apply. The AC should describe the Terraform-based approach:

• Option A: Set replicas = 0 in the Helm values within helm_release.ollama
• Option B: Add an ollama_enabled variable that conditionally includes the Helm release
• Option C: Add a count = 0 on the helm_release.ollama resource

The AC for nvidia-smi and graceful degradation are verifiable.

Blast Radius

• pal-e-docs semantic search -- acknowledged in the issue. Degrades to keyword-only. The project-pal-e-docs page confirms semantic search depends on Ollama (qwen3-embedding:4b). Graceful degradation claim should be verified.
• NVIDIA device plugin -- separate Helm release in same module, not affected by Ollama scale-down.
• Salt services state -- salt/states/services/init.sls already has ollama-service: service.dead (host systemd Ollama disabled). No conflict.
• Embedding worker -- if an embedding worker pod exists that connects to Ollama, it would fail on embedding requests. Issue does not address this -- needs AC for embedding worker behavior.

Decomposition Assessment

1 file target, 3 acceptance criteria, estimated agent work under 5 minutes. No decomposition needed.

Recommendation

• [BODY] Fix file target: replace "K8s Ollama deployment manifest" with terraform/modules/ops/main.tf (resource helm_release.ollama). Specify the approach: set replicas = 0 in Helm values or add count = var.ollama_enabled ? 1 : 0.
• [BODY] Fix AC #1: replace kubectl scale with Terraform-based verification (e.g., "tofu apply succeeds with Ollama scaled to 0" or "kubectl get deployment -n ollama shows 0/0 replicas after apply").
• [BODY] Add "Tests pass" to Checklist section per template.
• [SCOPE] Add gpu-freedom story entry to project-pal-e-world user-stories table (story note exists but is not registered on the project page).
• [SCOPE] Create architecture note arch-k3s for k3s component. Or relabel to arch:ollama if preferred (consistent with prior review review-1552-2026-06-22-r2).

Verdict: NEEDS_REFINEMENT

Template Completeness

Template: template-issue-bug

• [x] Type — Bug
• [x] Lineage — Standalone, discovered during Pal-E World Phase 2 testing
• [x] Repo — ldraney/hyprland-config
• [x] What Broke — Steam overlay pipe crash every 1-3 minutes on XWayland/Hyprland
• [x] Repro Steps — 4 clear steps provided
• [x] Expected Behavior — Present with 3 potential solutions listed
• [x] Environment — archbox, GTX 1070, Hyprland + XWayland, Steam stable
• [x] Acceptance Criteria — 3 criteria
• [x] Related — pal-e-world project referenced

All required Bug template sections are present.

Traceability

• [x] story:remote-play label — present on board item
• [ ] story note MISSING — project-pal-e-world user-stories section lists "pal-e-world" but no "remote-play" entry. [SCOPE] Add "remote-play" user story entry to project-pal-e-world user-stories section, or align the label to story:pal-e-world. Note: board items #1651 and #1654 also use story:remote-play, so adding the story entry is preferred.
• [x] arch:infra label — present on board item
• [ ] arch note MISSING — search for arch-infra returned zero results. [SCOPE] Create architecture note arch-infra for component infra.
• [x] Forgejo issue — ldraney/hyprland-config#14, state: open

File Targets

The issue body does not specify file targets. Bug template does not require them, but the fix location is identifiable from codebase inspection:

• [ ] hyprland.lua line 137 — ISSUE: Not mentioned in issue body. This is the Steam launch command (pgrep -x steam || steam) that needs a gamescope wrapper or -no-overlay flag. [BODY] Add as file target.
• [x] docs/architecture.md line 76 — verified: currently says crash is "Unfixable" — will need update if workaround succeeds
• [x] docs/gpu-sharing-gow.md line 5 — verified: currently says crash is "unfixable on Hyprland's XWayland" — same doc update needed
• [x] docs/pal-e-world.md lines 73-93 — verified: documents the crash and solutions, includes code snippet

Repo Placement

OK — issue is filed on ldraney/hyprland-config, fix is in hyprland.lua which lives in that repo. Single repo, no cross-repo concerns.

Dependencies

• #1651 "Install Sunshine on archbox" (3pt, story:remote-play) — same story, independent. Sunshine streaming is a separate concern from Steam stability.
• #1654 "E2E: MacBook to Moonlight to Tailscale to archbox" (2pt, story:remote-play) — soft dependency; E2E test benefits from Steam being stable, but not blocked.
• #1650 "Scale down k8s Ollama to free GPU" (1pt, story:gpu-freedom) — independent, different story.
• No items in in_progress that block this ticket. Phase 1 "Verify & Configure Server" is in_progress but unrelated.
• Dependencies are not documented in the issue body — acceptable for a standalone bug.

Acceptance Criteria

3 criteria — reasonable count.

• "Steam runs without crashing for 30+ minutes" — requires manual testing (agent cannot observe runtime stability). Testable but not automatable.
• "Working solution documented in this repo" — agent-verifiable (check for doc updates).
• "No regression in Hyprland session stability" — requires manual testing. Testable but not automatable.

Missing AC: docs/architecture.md and docs/gpu-sharing-gow.md currently claim the crash is "Unfixable." If a workaround is applied, these docs need correcting. [BODY] Add AC: "Update docs that claim crash is unfixable."

Blast Radius

• docs/architecture.md:76 and docs/gpu-sharing-gow.md:5 both state the crash is unfixable. A successful fix contradicts these docs.
• gamescope window rule already exists in hyprland.lua:158, indicating gamescope was anticipated but not wired into the launch path.
• The GOW container approach (docs/gpu-sharing-gow.md) provides an alternative path running Steam under native Xorg inside containers, avoiding this crash entirely. This fix is specifically for the native XWayland path.
• A worktree steam-crash-fix already exists, suggesting prior work was started. No conflict risk, but the implementer should check for stale state.

Decomposition Assessment

No decomposition needed.

• File targets: 1-2 files in 1 repo (hyprland.lua + docs)
• Acceptance criteria: 3 (under threshold of 5)
• Estimated agent work: under 5 minutes
• Points: 1 (correctly sized)

Recommendations

• [SCOPE] Add "remote-play" user story entry to project-pal-e-world user-stories section. Three board items (#1651, #1652, #1654) use this story label but no backing entry exists.
• [SCOPE] Create architecture note arch-infra for component infra. Multiple board items reference this arch label.
• [BODY] Add file target to issue body: hyprland.lua line 137 — Steam launch command needs gamescope wrapper or -no-overlay flag.
• [BODY] Add acceptance criterion: "Update docs/architecture.md and docs/gpu-sharing-gow.md to remove 'unfixable' claim if workaround succeeds."

story: GPU Freedom

Role

Developer / Gamer (Lucas)

Key

gpu-freedom

Want

As a developer who games on archbox, I want LLM inference offloaded from local Ollama to Groq API

So That

So that the GPU is free for Steam/Sunshine gaming without losing semantic search or platform LLM features

Acceptance Criteria

• Ollama scaled to 0 replicas in k8s, GPU free (nvidia-smi shows no Ollama)
• Groq API handles embedding generation for pal-e-docs semantic search
• Semantic search returns relevant results via Groq
• Cost stays under $1.00/day for normal platform usage
• Keyword search fallback still works as graceful degradation

Success Metric

GPU is fully available for gaming (0% Ollama usage) while semantic search quality is maintained at parity with local Ollama.

Related Architecture

No architecture notes yet for pal-e-world — this is the first story.

Related

• board-pal-e-world — project board
• pal-e-platform #486 — Scale down k8s Ollama
• pal-e-api #283 — Migrate semantic search to Groq API

story: Remote Play

Role

Gamer (Lucas)

Key

remote-play

Want

As a gamer, I want to stream games from archbox to any Moonlight client over Tailscale

So That

So that I can play from my MacBook Air, a library PC, or my phone — anywhere on the tailnet without needing local gaming hardware

Acceptance Criteria

• Sunshine installed and running natively on archbox with H.265 NVENC encoding
• Steam runs on workspace 9 without XWayland overlay crashes for 30+ minutes
• Moonlight on MacBook Air pairs with Sunshine via Tailscale IP
• Stream is stable at 1080p/60fps with < 50ms latency
• Audio streams correctly over Moonlight
• Connection survives 30+ minutes without drops

Success Metric

Can launch a game from MacBook Air via Moonlight and play a full session (30+ min) without crashes or unplayable latency.

Related Architecture

No architecture notes yet for pal-e-world — this is the first story.

Related

• board-pal-e-world — project board
• hyprland-config #13 — Install Sunshine
• hyprland-config #14 — Fix Steam XWayland crash
• hyprland-config #15 — E2E remote play test

Validation: #479 Vast.ai Bootstrap

PR

PR #481 merged to main via squash. CI pipeline #929 passed (clone step only — Salt-only changes, no tofu plan).

Deliverables

• docker-compose.yml.j2 — GOW 4-container stack (xorg, pulseaudio, sunshine, steam) with NVIDIA runtime, shared pulse-socket volume, persistent /workspace storage
• onstart.sh.j2 — Tailscale install + ephemeral auth, docker compose up, logging
• start.sh.j2 — Updated with template passing (docker_compose + onstart params), Tailscale MagicDNS resolution for Moonlight, session timeout watchdog
• init.sls — New states for compose template and onstart script deployment, dead tailscale-key state removed
• gaming.sls — Sunshine credentials, Tailscale hostname, Moonlight defaults, 100GB persistent disk
• secrets/gaming.sls — SOPS placeholders for sunshine_password and tailscale_auth_key

QA

Two independent QA passes (dev agent self-QA + dedicated QA agent). First pass found 3 blockers:

1. PulseAudio socket sharing broken (IPC namespace ≠ shared filesystem) — fixed with shared volume
2. Tailscale IP never discovered by client — fixed with MagicDNS hostname resolution
3. Sunshine password security tradeoff undocumented — documented

All blockers fixed in commit b33b1e8, CI pipeline #929 passed.

Deployment Note

Salt-only change. Requires salt '*' state.apply gaming-session on archbox to deploy rendered templates. SOPS secrets (sunshine_password, tailscale_auth_key) must be encrypted before first use — placeholders will trigger preflight check failures in start.sh.

Verdict

PASS — merged, CI green, QA blockers resolved.

Architecture: Helm

Helm is used for third-party chart deployments on k3s. Currently manages the Palworld dedicated server via twinki14/palworld-server-chart (v2.1.0).

Usage in Pal-E World

• Palworld dedicated server: Helm release in palworld namespace, CPU-only (no GPU)
• Config: Helm values set via OpenTofu helm_release resource in ldraney/palworld-server/terraform/main.tf
• Ports: UDP 8211 (game), TCP 25575 (RCON, deprecated), TCP 8212 (REST API, Palworld 1.0+)

Conventions

• Pin chart versions explicitly
• Set values via OpenTofu set blocks, not a values.yaml file
• Storage class: local-path
• Service type: ClusterIP (access via Tailscale, not public)

Vision

Pal-E World — a Palworld server with a real player economy, unhackable security, and no ceiling on growth. RuneScape-style guilds, manufacturing, trading, wars, and governance running on self-hosted infrastructure. Security is the product. Trust is the currency.

Projects & Repos Touched

Project/Repo	Platform	Role in this plan
forgejo_admin/palworld-server	Forgejo	Primary — server config, Terraform, mods (migrated from GitHub 2026-03-08)
forgejo_admin/pal-e-world-site (planned)	Forgejo	Svelte landing page — vision, signup, SEO. Standard service onboarding.
pal-e-platform	Forgejo	Host infrastructure (archbox k3s cluster)
pal-e-services	Forgejo	Landing page onboarding via var.services (Phase 3)

Context

This is the first plan for Pal-E World. The Palworld dedicated server already exists — deployed to k3s via OpenTofu Helm chart, with game client streaming (GPU + Sunshine/Moonlight). Repo migrated from GitHub to Forgejo on 2026-03-08.

Research completed 2026-03-08 confirmed:

• Palworld runs on UE5, modded via UE4SS (community framework, mature and widely used)
• Dedicated server is free, runs on Linux, Docker containers available (palworld-server-docker)
• Archbox meets hardware requirements (4+ cores, 16GB+ RAM, SSD)
• Lua mods can hook game functions: chat, pal systems, custom logic. Economy systems are buildable
• Base limit mods already exist (Unlimited Bases). World settings are fully configurable
• Steam handles player auth (SteamID64). One account = one player enforced by Steam
• No existing open source alternative — Palworld modding is the fastest path to playable

What's already done:

• [x] Vision documented in pal-e-docs (project-pal-e-world)
• [x] Open letter HTML created (palworld-vision.html)
• [x] Modding research complete — UE4SS, Lua, PalSchema, server hosting all viable
• [x] Palworld server deployed to k3s (Helm chart, PVC, RCON, auto-restart)
• [x] Game client streaming deployed (xorg + pulseaudio + sunshine + steam)
• [x] Repo migrated from GitHub to Forgejo
• [ ] Server verified running / brought back up
• [ ] Mods installed
• [ ] Landing page live

Previous Plan

None — first plan for this project.

Depends On

None. Archbox is already running. k3s cluster operational. palworld-server Helm chart already deployed.

Decisions Made

Decision	Rationale
Migrated palworld-server from GitHub to Forgejo	All active work on Forgejo. DORA tracking, Forgejo issues, Woodpecker CI, sprint boards. GitHub kept as archive remote.
Self-host on archbox via k3s (existing deployment)	Already deployed and proven. Free, full control, proves the self-hosted model. Uses thijsvanloef/palworld-server-docker Helm chart.
Platform-level deployment (not ArgoCD service)	Uses upstream Helm chart, not a custom image. Same pattern as Forgejo, Grafana. Own Terraform state.
UE4SS + Lua for modding	Only viable modding framework for Palworld. Supports server-side mods, function hooking, custom logic. Active community.
Svelte for landing page via standard service onboarding	Lucas's choice. Custom SvelteKit app fits var.services pattern (Woodpecker → Harbor → ArgoCD). Will eventually integrate login/auth.
Play first, mod second	Get Lucas and James playing immediately. Modding pipeline is Phase 2. Economy mods are Phase 4. Don't over-engineer before playing.
Steam auth sufficient for MVP	SteamID64 is unique, one-account-one-player enforced by Steam. Custom auth deferred.
Server name: Pal-E World	Brand name. Will be used for DNS, landing page, and server identity.

Phases

Phase 1: Verify & Configure Server

Slug: phase-2026-03-08-1-server-verify
Goal: Verify existing Palworld server is running on k3s. Configure server name to "Pal-E World", tune settings for fun co-op. Lucas and James playing tonight.
Owner: Dev agent
Repo: forgejo_admin/palworld-server
Forgejo Issue: TBD (created when phase starts)

Steps:

1. Check kubectl get pods -n palworld — verify server pod is running
2. If down, cd terraform && tofu apply -var-file=k3s.tfvars to bring it back up
3. Update server name to "Pal-E World" in tfvars
4. Tune PalWorldSettings.ini — increase base limits, worker pal limits, adjust difficulty for co-op with a 6-year-old
5. Verify RCON access works
6. Verify connection via Tailscale IP on UDP 8211
7. Update README with Pal-E World branding and connection instructions

Deliverables: (on completion)

Phase 2: Modding Pipeline

Slug: phase-2026-03-08-2-modding-pipeline
Goal: Install UE4SS and key mods on the server — unlimited bases, better world settings, quality of life.
Owner: Dev agent
Repo: forgejo_admin/palworld-server
Forgejo Issue: TBD

Steps:

1. Install UE4SS framework on the dedicated server container
2. Install and configure mods:
   • Unlimited Bases (remove 4-base cap)
   • Increased worker Pals
   • Increased world settings limits
   • Any quality-of-life mods from Nexus that enhance co-op
3. Document mod installation process in repo README
4. Test that mods work on Linux dedicated server (some are Windows-only — verify compatibility)
5. Create a mod list manifest (mods.json or similar) for tracking installed mods and versions

Deliverables: (on completion)

Phase 3: Landing Page

Slug: phase-2026-03-08-3-landing-page
Goal: Ship a Svelte landing page that sells the vision and collects email signups. Onboarded as standard service via pal-e-services.
Owner: Dev agent
Repo: forgejo_admin/pal-e-world-site
Forgejo Issue: TBD

Steps:

1. Create pal-e-world-site repo on Forgejo
2. Scaffold SvelteKit app with Dockerfile and Woodpecker CI pipeline
3. Build landing page sections from the vision doc:
   • Hero: "The Palworld We Could Build Together" — the pitch
   • Unlimited Growth — base networks, progression to the stars
   • Real Economy — guilds, manufacturing, trading, wars, mercenaries
   • Unhackable Security — the big sell, trust-rated servers
   • The Prayers System — community voice and governance
   • Game Culture — serious, competitive, organic
4. Email signup form — collect email, store in DB (or simple file/API for MVP)
5. SEO basics: meta tags, Open Graph, structured data
6. Onboard via pal-e-services: add to var.services (Harbor project, ArgoCD app, Tailscale funnel)
7. Deploy and verify public access

Deliverables: (on completion)

Phase 4: Economy Mod POC

Slug: phase-2026-03-08-4-economy-mod
Goal: Build a proof-of-concept Lua mod that adds basic trading and guild economy to the Palworld server.
Owner: Dev agent
Repo: forgejo_admin/palworld-server
Forgejo Issue: TBD

Steps:

1. Set up Lua mod development environment with UE4SS
2. Build POC features:
   • Player-to-player item trading via chat commands
   • Basic coin/currency system (earn from selling, spend at player shops)
   • Guild registration and membership
   • Simple guild bank (shared storage)
3. Server-side persistence — save economy state across restarts
4. Test with 2+ players on the server
5. Document the mod API for future extension (missions, bounties, wars)

Deliverables: (on completion)

Key Files

Phase	File	Repo	Change
1	terraform/k3s.tfvars	palworld-server	Server name, settings
1	README.md	palworld-server	Pal-E World branding, connection instructions
2	mods/	palworld-server	UE4SS + mod files
3	src/routes/+page.svelte	pal-e-world-site	Landing page
3	terraform/services.tfvars	pal-e-services	Add pal-e-world-site to var.services
4	mods/PalEconomy/main.lua	palworld-server	Economy mod entry point

Verification

• [ ] Phase 1: Lucas and James can connect to the server and play together
• [ ] Phase 2: Server runs with unlimited bases mod active, no crashes
• [ ] Phase 3: Landing page loads at pal-e-world domain, email signup works, deployed via ArgoCD
• [ ] Phase 4: Two players can trade items via chat commands on the server

Next Plan Seeds

• Prayers system mod — in-game appeals channel, community voting, harassment reporting
• Guild wars mod — territory claiming, resource wars, alliance systems
• Trust rating system — server security scoring, public dashboard
• Multi-server federation — economy bridges between trusted servers
• Auth integration — pal-e-auth for landing page login, linking Steam accounts to web accounts
• Interplanetary expansion — research what's needed for new worlds (likely requires standalone game or deep UE5 modding)
• GPU pricing model — if offering cloud gaming, cost model to cover GPU compute. Terraform + AWS/Hetzner for scaling nodes in k8s

Related

• project-pal-e-world — project page with full vision and core concepts
• project-pal-e-platform — host infrastructure (archbox, k3s, Tailscale)
• project-pal-e-services — service onboarding (ArgoCD, var.services)
• service-onboarding-sop — how to add a new service to pal-e-services
• palworld-vision.html — open letter to Pocketpair and community

Plan: Headless Game Streaming Server

Vision

A fully headless game streaming server — no monitor, no dummy plug, just a GPU in a k3s cluster serving Palworld to any Moonlight client on the tailnet.

Projects & Repos Touched

Project/Repo	Platform	Role in this plan
forgejo_admin/palworld-server	Forgejo	All IaC changes (xorg config, deployment spec)

Context

The palworld-server repo was built while a physical LG UltraFine monitor was connected to the GTX 1070. The game client streaming pod (Phase 2) used Sunshine/Moonlight for remote play but assumed Xorg could find a real display. Now the server is headless — no monitor, remote-only access. The NVIDIA proprietary Xorg DDX refuses to create display outputs without a real display handshake, producing NULL metamodes. Sunshine then fails to capture or encode.

What's already done:

• [x] Switched xorg config from nvidia DDX to modesetting driver
• [x] Modesetting finds DP-2 (cable still connected, monitor off) and sets 1920x1080
• [x] Sunshine starts with libx264 software encoding, KMS capture works
• [x] Dedicated server scaled back to 1 replica and running
• [ ] End-to-end Moonlight connection not tested
• [ ] No NVENC hardware encoding (modesetting doesn't expose it to Sunshine)
• [ ] Depends on DP-2 cable being physically present

Previous Plan

None — first plan for this project.

Depends On

None.

Decisions Made

Decision	Rationale
Use modesetting Xorg driver instead of nvidia DDX	NVIDIA DDX produces NULL metamode without physical display handshake. Modesetting uses DRM/KMS directly, which detects DP-2 as connected via hot-plug detect pin in cable.
Accept software encoding (libx264) for now	Gets streaming working. NVENC requires either nvidia DDX (broken headless) or a dummy plug. Phase 2 addresses this.
Keep EDID configmap in place	Not currently used by modesetting (it reads real EDID from DP-2), but useful for future true-headless work with dummy plug or virtual display.
Forgejo is primary platform, GitHub is mirror	Origin remote is Forgejo. GitHub has legacy issues.

Phases

Phase 1: Headless display fix (modesetting driver)

• Goal: Get Xorg + Sunshine working without a powered-on monitor
• Owner: Main session
• Repo: forgejo_admin/palworld-server
• Forgejo Issue: forgejo_admin/palworld-server #1
• Status: In progress (code done, PR needed)
• Steps:
  1. Switch xorg configmap from nvidia driver to modesetting driver
  2. Add EDID configmap + volume mount (for future headless use)
  3. Verify Sunshine starts with working encoder
  4. Commit, PR, merge

Phase 2: Hardware encoding (NVENC)

• Goal: Replace libx264 software encoding with NVENC for lower latency and CPU usage
• Owner: Main session
• Repo: forgejo_admin/palworld-server
• Status: Not started
• Options:
  1. Dummy DP plug ($5) — makes GPU think monitor is connected, nvidia DDX works, NVENC available
  2. Investigate newer GoW/Wolf images with native headless + NVENC support
  3. Hybrid: modesetting for display + direct NVENC access for encoding

Phase 3: True headless (remove DP-2 cable dependency)

• Goal: Streaming works with zero cables connected to GPU
• Owner: Main session
• Repo: forgejo_admin/palworld-server
• Status: Not started
• Current modesetting approach only works because DP-2 cable is plugged in. Need either dummy plug or software virtual display.

Phase 4: Documentation update

• Goal: README reflects remote-only workflow, no host package references
• Owner: Main session
• Repo: forgejo_admin/palworld-server
• Status: Not started
• Steps:
  1. Rewrite Phase 2 section for Moonlight remote connection
  2. Add first-time setup guide (Moonlight pairing, Steam login)
  3. Remove host package references (xorg-server, dwm, moonlight-qt)
  4. Document Sunshine web UI access for pairing

Key Files

Phase	File	Repo	Change
1	terraform/main.tf	palworld-server	xorg configmap: modesetting driver, EDID volume
4	README.md	palworld-server	Remote-only docs, remove host package refs

Verification

• [ ] Phase 1: kubectl logs -c sunshine shows encoder found, no Fatal errors
• [ ] Phase 1: kubectl logs -c xorg shows valid mode (not NULL)
• [ ] Phase 2: Sunshine logs show NVENC/nvenc encoder
• [ ] Phase 3: Works with all cables unplugged from GPU
• [ ] Phase 4: README has no references to host packages
• [ ] End-to-end: Moonlight connects from laptop, Steam visible, Palworld launches

Next Plan Seeds

• Game auto-launch: Start Palworld automatically when Moonlight connects
• Save backup automation: Scheduled world save backups to S3/MinIO
• Multi-game support: Generalize client pod for other Steam games

Related

• project-pal-e-world — project page

Pal E World Board

Goal: Verify existing Palworld server is running on k3s, rename to "Pal-E World", tune settings for co-op, confirm Lucas and James can connect.

Owner: Dev agent

Repo: forgejo_admin/palworld-server

Depends on: None

Forgejo Issue: forgejo_admin/palworld-server #2

Scope

• [x] Verify palworld-server pod is running (CONFIRMED: 1/1 Running)
• [x] Clean up stale pods (deleted UnexpectedAdmissionError client, failed daily-reboot job)
• [x] Update server name to "Pal-E World" in k3s.tfvars.example
• [x] Update README with Pal-E World branding
• [x] PR submitted: forgejo_admin/palworld-server #4 — QA approved, awaiting merge
• [ ] Update local k3s.tfvars with server_name = "Pal-E World"
• [ ] tofu apply to push name change to cluster
• [ ] Verify RCON access
• [ ] Verify connection via Tailscale IP on UDP 8211
• [ ] Tune PalWorldSettings.ini for co-op (base limits, worker pals, kid-friendly difficulty)

Status

PAUSED. PR #4 open and approved, awaiting merge. Remaining items are manual (tofu apply, settings tuning, connection test). Paused to avoid conflicting with other palworld-server work Lucas is doing.

Discovered Issues

• Old client pod stuck in UnexpectedAdmissionError — RESOLVED (deleted)
• Failed daily-reboot jobs — RESOLVED (deleted)

Deliverables

• PR #4: README rebrand + k3s.tfvars.example server name (approved, not yet merged)

Related

• plan-2026-03-08-pal-e-world-server — parent plan
• project-pal-e-world — project page