Relayer 2.29.1

What's new in 2.29.12026-11-03

🔒 Security

• A bucket policy naming an account root now admits only that account
• Lock headers on an upload now need the lock permissions
• A multipart upload id only works on the bucket and key it was created for
• A signed upload is now checked against the body hash it was signed with
• Re-pointing an identity provider at a new `jwks_uri` takes effect at once
• A credential in an identity provider URL's query string is no longer written to the log
• HostIO refuses to start on a fresh install when its master secret is empty or shorter than 32 bytes
• MCP tools and resource reads now act only on the caller's own storage
• Anonymous requests to a `public-read` bucket can no longer read its ACL, policy, encryption or other configuration
• A bucket policy with a malformed `NotIpAddress` range now denies the request
• PutBucketLogging no longer accepts a target bucket whose owner lookup failed
• Audit and event-notification webhooks refuse unspecified and carrier-grade NAT targets

🛠 Fixes

• Lifecycle expiry is metered once, by the sweep
• Storage already over-metered is corrected once
• The costlog writer no longer delays start-up
• The costlog `method_name` type gains `LifecycleExpire` and `LifecycleExpireCorrection`
• A GET whose row fails to download gets the error instead of waiting forever
• `HOSTIO_CACHE_ONLY_READ` works again
• Idle background uploads no longer re-tune the learned upload line down
• Stalled upload sockets no longer hold a burst over the upload cap
• Upload cap tests no longer fail on a loaded machine
• A lifecycle rule scoped by a prefix and a tag no longer applies to the whole bucket
• Lifecycle expiration of an older object in a tenant bucket now deletes its stored data
• A tagging request with a `urn:uuid:` versionId answers 404 instead of 500
• The owner upload cap now holds the line rate, headers and resent pieces included
• A metered S3 request no longer waits for a rebuild of the whole billing total
• The audit-log Loki no longer crash-loops after a burst of log traffic
• A lifecycle AbortIncompleteMultipartUpload rule now aborts stale uploads in a tenant's bucket
• A retention or legal-hold request with a malformed `versionId` now gets 400
• The first start after an upgrade no longer keeps a database connection checked out
• A delete no longer panics when Postgres refuses to start a transaction
• A cache wipe without `force` no longer discards data that has not reached the providers
• A transient database error no longer makes HostIO upload a committed row again
• A failed cache write while saving a downloaded row no longer leaks cache space
• Stopping HostIO during the encryption migration no longer corrupts the moved files
• HostIO no longer allocates a fresh buffer on every read of an upload body
• The upload scheduler no longer walks the whole cache while holding the cache lock
• Recording an upload part no longer scans the whole parts ledger
• Concurrent billing writes no longer abort each other with a deadlock
• The box owner's bucket policy edits on a tenant's bucket now change that bucket's policy
• A tenant admitted by a bucket policy can list and tag objects in the box owner's bucket
• Rotate Credentials now changes the owner's S3 key
• Importing a second owner credential is refused
• Expired multipart uploads of suspended, deleted or overdrawn tenants are cleaned up
• Large buckets from before the upgrade no longer lose one object per 1,000 to the cleanup sweep
• Space held by a bucket whose stored entries are all leftovers is now released
• Keys written again right after a big DeleteObjects call now stay in listings
• HostIO keeps compat uploads under the owner's upload cap
• Deleting an old version of an object no longer makes the object vanish from bucket listings
• Objects hidden from bucket listings by the earlier version-delete bug are listed again
• A full disk, busy write slots and an unreachable billing service now reach S3 clients as a retryable 503 instead of a 500
• DeleteObject with a version id no longer deletes the live object on a bucket that never had versioning
• GET `?retention` on a version without retention answers 404 NoSuchObjectLockConfiguration
• A bucket default retention above about 292 years no longer sets retain_until in the past
• GET, HEAD, GetObjectAttributes, DELETE and DeleteObjects with a version id that is not a UUID answer 400 InvalidArgument instead of 500
• CreateBucket answers 503 ServiceUnavailable instead of 500 when the gateway backend refuses the connection
• A key whose policy uses AWS's Object Lock configuration action names can now read and set a bucket's lock settings
• HostIO no longer crashes when a host rejoins while uploads to it are still in flight
• HostIO stops waiting for the gateway as soon as shutdown starts
• Per-host data statistics keep their changes when a database write fails
• Errors from sector downloads by token now name the call that failed
• Presigned URLs from the MCP server now verify for keys containing `+`, `=`, `@` and similar characters
• The MCP `put_object` tool now rejects a call without `content`
• Copying an object no longer loses the usage records batched with it
• A cost center with an old pricing rule no longer fails every usage insert
• Storage is charged for the exact time elapsed
• The usage consumer no longer panics when Kafka closes a partition claim, and the usage publisher no longer spins after its producer closes
• Partition-integrity repair works for a cost center whose name contains a slash
• Gateway shutdown closes the cost-log client
• AbortMultipartUpload checks the upload ID before interrupting the upload,
• Freshly installed gateways now get the `idx_files_covering` index that ListObjects reads
• Deleting a file while its upload is being finalized no longer crashes HostIO or leaves the cache locked
• A download canceled at the wrong moment no longer leaks cache space
• Partial cache eviction keeps the part of a piece that touches its left neighbor
• The upload planner no longer produces a negative-length upload range
• `CacheStorage.Read` reports the bytes it read to the memory manager
• ListBuckets no longer fails with `column "content_type" does not exist` on a box whose metadata tables lack the column
• An OIDC provider whose issuer identifier ends in a slash now works instead of failing every federated login
• A multipart completion rejected with `InvalidPart` no longer finishes the upload or counts its bytes as stored

⚙️ Changes

• An upload dial the owner sets is pinned; an unset one is autotune's
• The canned Backup policy now allows s3:DeleteBucket
• The canned ReadOnly policy now allows the Object Lock reads

✨ New

• HostIO learns the upload line from its own uploads and sets its upload dials
• HostIO follows a changed line
• The found line survives a restart
• Health reports what autotune decided
• HostIO measures its download rate and learns the download line
• HostIO sets its download workers from memory
• Owner keys `HOSTIO_AUTOTUNE_OWNER_CAP_MBPS`, `HOSTIO_AUTOTUNE_BACKOFF`, `HOSTIO_AUTOTUNE_LINE_MBPS` and `HOSTIO_AUTOTUNE_SHARE_PCT`
• The back-off switch caps uploads from the first byte, and hostio says when the CPU, not the line, sets the upload speed
• HostIO adjusts its upload rate to the line while it runs
• The upload limiter always exists, so the rate can change without a restart
• On Linux, upload sockets keep at most 64 KiB unsent while the loop is on
• A built-in `Backup` policy lets a scoped key run immutable backups
• Muse has a fourth caller, `readonly`, for staff and reporting reads

🗑 Removed

• Muse no longer has a `max_contract_locked_funds` setting

🔒 Security

• A compressed request body with trailing bytes no longer crashes the server
• Image builds no longer print the npm registry token
• The Relayer API accepts only access tokens
• Browser sign-in sends a PKCE challenge
• The HostIO secret is generated with a secure random source

🛠 Fixes

• Restoring a backup while the setup check or a cache relocation was running showed "Restore failed — check connectivity a
• Choosing HTTP Only on the Certificates page took the S3 gateway down: S3 on:9000 and:9443 stopped answering and stayed
• When the bucket list failed to load, File Manager showed "0 BUCKETS" and "No bucke
• A prefix-only lifecycle rule lost its prefix when Bucket Settings read it back, because the gateway returns that rule wi
• Adding or deleting a lifecycle rule failed with "Each lifecycle Filter.Tag must include a non-empty string Key" whenever
• Bucket Settings offered Abort Incomplete Multipart on a tag-scoped rule, which the gateway refuses, and showed the refus
• A backup restore whose database step failed kept going: it wrote the config, HostIO keys and Samba files from the backup
• Nothing opened the Relayer Bug Report dialog: its `submitBug` listener in `MainTheme.vue` had no sender, so users could
• `POST /api/v1/system/bug-report` never answered when the Discord send failed or building the report threw, so the reques
• A takeover the console refused still moved the box:
• The bucket summary, the tenant dashboard and the tenant usage view said "Billed on the network total at $4.00 per TB per
• On Provider Profiles, choosing a continent or an XNS Region in the pre-select filled Data View, but the banner still rea
• Restart All and Apply Changes on `/configuration` now show the please-wait overlay while the services restart, as Backup
• Escape now does what a dialog's Cancel or close control does in the dialogs that ignored it: backup restore, the page mo
• The share groups list showed the next day as "Created" for a group made in the evening west of UTC, and "Created Invalid
• The design-system base reset now loads. Buttons, inputs, selects and text areas now also inherit
• A restore no longer leaves the HostIO encryption keys and the HostIO secret out of step
• HostIO, Gateway and S3Gateway start again after an owner reset
• A cache wipe can no longer silently delete uploads that have not reached providers
• The image is built on a Node 22 that still gets security fixes
• The dashboard no longer shows a speed that disagrees with other readings
• A UI restart no longer restarts HOSTIO over the derived upload values
• Stop keeps HOSTIO stopped during a restart the UI started
• Issuing a Let's Encrypt certificate saves the SSL mode and answers the page
• The Cloudflare token no longer reaches the S3 gateway's environment
• Let's Encrypt certificates renew again
• A bucket-scoped read role can read Object Lock settings
• Pages no longer sign the owner out while an upload fills the line
• A failed key fetch logs its reason
• The start-up card no longer hangs silently on "Checking authentication"
• Reconnect on the credential-revoked banner opens the first-access dialog
• Provider Profiles no longer hangs on "Please wait." when a region zone has no name
• Provider Profiles loads before the first provider fetch has succeeded
• Network Overview and Provider Profiles download the provider catalog once per poll, not twice
• Settings saves no longer fail with "Could not save configuration." when the provider list was never fetched
• The object detail panel no longer logs about 20 errors on a bucket without Object Lock
• The Tags tab of a deleted file says the file was deleted
• Creating a bucket no longer reports auto-sharing as enabled when it is off
• A failed upload no longer leaves a partial file in the upload spool
• The share users list shows "Created —" for a user with no creation date
• Attach Policy reports a failed request and no longer reports a false success
• Throughput chart y-axis labels are no longer cut off
• The Not Found page is styled again
• The Transport choice card no longer covers the theme toggle
• The sidebar reads as one gray surface in light theme
• A certificate and key that do not match no longer take the dashboard down
• Two operators editing the same bucket's grants no longer silently overwrite each other
• Certificate upload accepts EC private keys and keeps the intermediate chain
• A certificate installed after start reaches the S3 HTTPS port without a container restart
• The share users list shows the day a user was created in the browser's own time zone
• Server test coverage can be measured again
• demo-backend's provider cache test TP-33 no longer fails under load
• demo-backend's single-upload memory test TP-35 no longer fails on normal RSS noise
• Choosing an SSL mode on the Certificates page saves again
• SSL mode changes are refused on hosts that manage their own certificate
• "No SSL" is refused while "Encrypted-only storage" is on
• `GET /api/v1/config/ssl` answers 200 `{ success: false, installed: false }` when no certificate is installed, not 405

⚙️ Changes

• The setup banner speaks to the owner's answers
• Checkboxes take their check and dash marks from `@scpcorp/design-system` 0.3.3
• `PUT /api/v1/config` answers 409 `settings_changed` when the body carries a `Version` other than the stored one, so a pa
• Upload workers, download workers, uploads in flight and the sector upload timeout are owner overrides: empty by default
• One-time upgrade pass
• The Workload select offers "Standard" and "Bulk archive ingest". Bulk archive ingest no longer sets `HOSTIO_UPLOAD_WORKE
• Settings pages that end in a setup check reload the server-derived HostIO values, keeping unsaved edits
• CI: the non-gating `s3_tool_latest` job is gone from merge request pipelines. It warned on nearly every merge request wh
• `@scpcorp/design-system` 0.3.2. In the light theme the sidebar is now gray (`--ds-bg-sidebar` `#dde1e8`, wa
• In light, the readers that lost their edge against the gray get one, in their own component styles: a 1px `--ds-text-dim
• Inherited from design-system 0.3.1: dialog titles have no default margin. Tailwind's

✨ New

• The sidebar now has a Support entry under Account that opens the XNS Console ticket desk (`https://console.xns.tech/my-t
• The Speed tab asks the owner two setup questions
• While HostIO runs, the Speed tab says "Saving restarts HostIO. Uploads in progress are interrupted." beside Save answers
• `POST /api/v1/system/setup-answers` writes the answers to the stored `HostIO` section and re
• First boot sizes the cache from free disk space
• `GET /api/v1/system/cache` carries HostIO's `autotune_stats` block under `HostIO`, and none while HOSTIO is stopped. The
• The bridge exporter carries HostIO's upload autotune
• A setup check on a fresh install checks that the sign-in and provider hosts answer
• The check's result is stored as `SetupCheck` in `relayer_configuration.json` and served on the sign-in-free `GET /api/v1
• `POST /api/v1/system/setup-check` re-runs the check. It ne
• A setup banner on every page shows only when the check found a problem, with the
• Configure has a Speed tab
• The NOC Network card has an Upload row with the learned upload speed, updated at each 30 s poll
• Tenants can pick a `Backup` key template for immutable backups

🗑 Removed

• The Persona preset bar in Advanced Settings and the "Slow Upload Connection" preset
• Upload profiles are gone
• The UI no longer derives a simultaneous-upload ceiling or an upload cap: the start-up reconcile that wrote them and the
• The Relayer no longer runs an internet speed test

Previous releases

🛠 Fixes

• A tenant who runs out of funds is now blocked, and one who tops up is unblocked, within one poll interval instead of at the next costlog restart

⚙️ Changes

• The gateway request-rate defaults now match what customer boxes run

✨ New

• `HOSTIO_MAX_UPLOAD_MBPS` limits how fast hostio sends sector data to hosts

🛠 Fixes

• A box whose storage account and owner account are the same name no longer gets stuck restarting forever after an upgrade

⚙️ Changes

• The packer's flush defaults are now 97% / 10m, so rows ship nearly full by default

🔒 Security

• s3gateway: `s3:*` no longer authorizes non-s3 actions — closes a namespace-crossing privilege escalation exposed by the IAM self-service reversal

⚙️ Changes

• mt-reshape E-A2: the tenant byte quota lives on the cost-center, not on `s3_tenants`

✨ New

• The S3 Gateway's multi-tenant conformance gaps close, and the published compatibility number becomes reproducible instead of hollow
• Every storage snapshot tick now lays down the box's aggregate row byte-identically PLUS one row per tenant — the source data finally tells the per-tenant truth

🛠 Fixes

• monitoring: WORM-defeat alerts missed their ≤60s detection SLA on the default Alertmanager route
• s3gateway: valid credentials were rejected with `InvalidAccessKeyId` under high request concurrency

✨ New

• s3gateway: runtime Sentry kill-switch — silence error reporting without a rebuild
• monitoring: alert delivery spine — firing rules now reach the operator instead of evaluating into a void
• monitoring: Watchdog dead-man's-switch — the operator is told when the monitoring stack itself dies
• monitoring: alert thresholds are `.env`-tunable — threshold changes are a config edit + recreate, never a source change or rebuild
• docs: observability & alerting operator runbook
• s3gateway: audit-pipeline health metrics registered unconditionally + NDJSON write-health counters
• s3gateway: WORM-defeat attempt detection — counters + ≤60s trip-wire alert
• monitoring: application logs now reach Loki — the operator can search every service's logs, not just audit
• hostio + s3gateway: cache-hit-rate and per-bucket activity are now instrumented — the two series the AI-engineer overview screen needs

🛠 Fixes

• s3gateway: a configured rate limit never throttled the box owner's account or any of its service-account/IAM keys
• s3gateway: service-account create/delete and credential rotation left no audit trail
• s3gateway: clients uploading an object could see the connection reset instead of the real error response
• s3gateway: running out of physical disk returned a generic internal error instead of a storage-full response
• s3gateway: newly created users and access keys could be assigned a cost center that doesn't exist, leaving them unable to perform any storage operation; such requests are now rejected up front with a clear error
• s3gateway: deleting a bucket and immediately recreating it under the same name could fail until a brief background cleanup finished

✨ New

• audit landing: `S3GW_AUDIT_LOKI_PUSH_URL` lets a bring-your-own-Loki admin redirect audit logs to an external Loki with one env var — no UI, no rebuild of behavior

🔒 Security

• SEC-1: drop `127.0.0.1:9101:9101` host publish from `docker-compose.relayer.yml`. SEC-2: set `KEYCLOAK_AUDIENCE=relayer`

🛠 Fixes

• s3gateway: conditional writes on a missing object no longer 500
• gateway + s3gateway: streaming write errors no longer destroyed by TCP resets; internal-hop outages now retryable 503s
• hostio: delete-during-upload no longer crash-loops HOSTIO
• s3gateway: DeleteBucket no longer `500`s on large buckets, and previously wedged buckets heal themselves
• s3gateway: large/slow uploads no longer fail with a 60s `500 InternalError`
• s3gateway: 5xx Sentry events are now self-diagnosing and no longer fan out per object key
• s3gateway: delimited `ListObjects` no longer returns phantom empty folders after a versioned delete
• s3gateway: `AssumeRoleWithWebIdentity` now resolves multi-audience tokens correctly
• s3gateway: `PutBucketPolicy` now rejects `NotPrincipal` on `Allow` statements with 400 `MalformedPolicy`
• s3gateway: temp credentials now enforce `X-Amz-Security-Token` on every signing path
• s3gateway: bucket-policy `Principal` is now enforced
• Monitoring stack now survives install/upgrade/reboot
• s3gateway: bootstrap owner credential is now seeded with `is_owner=true`
• s3gateway: UploadPartCopy returns `InvalidRange` for out-of-bounds copy range
• gateway: versioned GET of small multipart-completed objects no longer returns HostIO fileID as content
• hostio/cacherepo: removed diagnostic logging

⚙️ Changes

• s3gateway: reword four stale `:9000` source comments

✨ New

• hostio: read-only `POST /v1/hostio/evaluate` endpoint
• all services: per-service structured logging bootstrap + support-grade operational stream
• s3gateway: `AccessLogWorker` — bucket-logging access-log delivery
• s3gateway admin: per-bucket live-object stats endpoint `GET /admin/v1/bucket-stats`
• hostio: bounded upload host-switch loop
• hostio + s3gateway: staging load-shed → S3 503 SlowDown with Retry-After
• gateway + s3gateway: client-disconnect classified as a clean disconnect, not a 500
• hostio: Go-side backstop clamp on `SimultaneousUploads`
• docs: version → commit index + 2.0.0 promotion note
• s3gateway: webhook audit sink — signed NDJSON audit export to a customer SIEM
• s3gateway: AWS IAM query-protocol dialect — 16 user-plane management verbs for Terraform/Vault
• s3gateway: per-request audit-record enrichment — one record answers both "who/allowed?" and "how big/how long/what client/what error?"
• s3gateway: pluggable audit sinks + opt-in Loki/Alloy audit landing
• s3gateway: cross-account `AssumeRole` — persisted roles, trust policies & `ExternalId` confused-deputy guard
• s3gateway: `AssumeRoleWithWebIdentity` — keyless OIDC federation for ephemeral workloads
• s3gateway: dual-listen — `:9000` is now permanently plain HTTP, HTTPS is additive on `:9443`
• s3gateway: admin/metrics `:9101` decoupled from the S3 TLS cert — now always plain HTTP
• s3gateway: in-account canonical-ID unification — IAM/SA/STS delegation now works end-to-end
• s3gateway: Sentry error reporting
• Observability dashboards + baseline alerts
• Monitoring sidecar stack