SOP: Harbor Robot Import Recovery

sop-harbor-robot-import Sop

sop active

Project

SOP: Harbor Robot Import Recovery

Purpose

Use when tofu apply on pal-e-services fails with a Harbor 409 conflict on a harbor_robot_account create — the robot already exists in Harbor but is not in terraform state. Applies to platform operators (human or agent) recovering from partial applies, drift, or out-of-band robot creation. The outcome is a clean tofu plan with the existing live robot correctly mapped into terraform state, with no risk of marking an unrelated live robot for replacement.

Background

The Harbor 409 error format invites a critical misread. Example error:

The number before the colon (27) is the Harbor project ID, NOT the robot ID. The format is <project_id>:<robot_full_name>. Importing /robots/27 based on this misread will likely point terraform state at an unrelated robot, and the next plan will mark that robot for must be replaced, threatening live image-pull access for an unrelated service. Verified incident: 2026-04-26, near-miss on westsidekingsandqueens-pull.

Compounding the trap: GET /api/v2.0/robots?page=N&page_size=100 returns system-level robots only. Project-scoped robots (which is what every service uses for CI/pull) are filtered out. Direct GET /api/v2.0/robots/{id} lookups are the only reliable cross-cutting method.

Steps

Extract the robot full name from the 409 message — the part after the colon. From 27:playme2k+playme2k-ci the full name is playme2k+playme2k-ci. The Harbor record's stored name field will be robot$<project>+<robot-name>, e.g. robot$playme2k+playme2k-ci.

Find the real robot ID by direct ID scan against /api/v2.0/robots/{id}. Do NOT use the paginated listing endpoint. Run:

HARBOR_PASS=$(grep -E '^harbor_admin_password' ~/pal-e-services/terraform/k3s.tfvars | sed 's/.*= *"\(.*\)".*/\1/')
TARGET="playme2k-ci"  # adjust to the robot name from step 1

for id in $(seq 1 400); do
  r=$(curl -s -u "admin:$HARBOR_PASS" "https://harbor.tail5b443a.ts.net/api/v2.0/robots/$id")
  name=$(echo "$r" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('name','')) if isinstance(d,dict) and 'errors' not in d else None" 2>/dev/null)
  if [[ "$name" == *"$TARGET"* ]]; then echo "$id $name"; fi
done

Expected output: a single line with the real robot ID and full name (e.g. 277 robot$playme2k+playme2k-ci). Adjust the seq upper bound if more robots have been created since this SOP was written.

Import with the verified ID. From ~/pal-e-services/terraform:
```
tofu import -var-file=k3s.tfvars 'harbor_robot_account.service_ci["<service>"]' '/robots/<real-id>'
```
Use service_ci or service_pull matching the resource block in services.tf. Expected success signal: Import successful! The resources that were imported are shown above.
Verify safety with tofu plan immediately. This is the safety gate. Run:
```
tofu plan -lock=false -var-file=k3s.tfvars 2>&1 | grep -A 5 'harbor_robot_account.service_ci\["<service>"\]'
```
Expected output: in-place updates only (e.g. label diffs) or no diff at all. If the output contains any of must be replaced, ~ name = "..." -> "...", or -/+ resource, the wrong robot was imported. Stop and proceed to the Recovery section before running any apply.
Apply only after step 4 is clean. Run tofu apply -var-file=k3s.tfvars per the standard sop-platform-tf-changes workflow.

Recovery

If step 4 reveals the wrong robot was imported, immediately back out the bad mapping. tofu state rm only edits terraform state — it does NOT touch the live Harbor robot:

Confirm the live robot is unaffected (kubectl get pods in the affected service's namespace should show no ImagePullBackOff). Return to step 2, scan a wider ID range, and try again. Do not run tofu apply until the import → plan loop produces clean output.

Rules

Never treat the number in a Harbor 409 message as a robot ID. It is the project ID. Always look up the real robot ID before importing.
Never use GET /api/v2.0/robots?page=N to find project-scoped robots. The endpoint silently filters them. Use direct ID lookups.
Always run tofu plan immediately after every import. If the plan shows replacement or name change for the imported resource, it is wrong — back out before doing anything else.
Never run tofu apply with an unverified import in state. The blast radius can include destruction of unrelated live infrastructure.
tofu state rm is safe to run in this recovery context — it edits only the local state file and never touches Harbor or Kubernetes resources.
If a robot truly does not exist in Harbor (verified via direct ID scan returning no matches), the create is legitimate — the 409 came from a different cause and this SOP does not apply.

service-onboarding-sop — happy-path service onboarding that creates Harbor robots from scratch
sop-platform-tf-changes — overall terraform change workflow this SOP plugs into
sop-incident-response — broader incident playbook for cases where this recovery alone is insufficient
feedback_never_alter_prod_directly — underlying principle: verify before mutating
feedback_never_guess_state — never assert system state without verification

SOP: Harbor Robot Import Recovery