Nice. I also ran a small Colab experiment on my end:
Using a fresh Colab Free runtime, I could reproduce a no-progress state while downloading the same model.safetensors from juiceb0xc0de/bella-bartender-gemma-e4b.
The most useful result was not just that it stalled again, but that the same file passed the reported 2 GB region when I explicitly disabled Xet.
This makes a simple explanation such as “the model file is physically truncated at 2 GB” much less likely.
That is still a black-box inference, not a confirmed internal root cause.
For someone who only needs to retrieve the file, the most direct diagnostic fallback is:
Colab reproduction, ablation matrix, and interpretation
Test setup
I used a fresh Colab Free runtime with:
- no Hugging Face token;
- public repositories only;
- a separate empty
HF_HOME, Hub cache, Xet cache, and destination for every branch;
HF_DEBUG=1;RUST_LOG=info;RUST_BACKTRACE=full;HF_XET_LOG_FILE set for each run;- destination, Hub-cache, Xet-cache, console-log, and Xet-log growth monitored every ten seconds;
- a watchdog that stopped a branch after four minutes with no observable change;
- a threshold test that intentionally stopped a branch after it materialized more than 2.5 GB.
The 2.5 GB threshold was chosen only to answer:
Can this branch pass the repeatedly reported approximately-2-GB stopping region?
It does not mean the entire 15.9 GB file completed.
The official Xet repository provides diagnostic wrappers for hangs and unexpected behavior, including stack snapshots and debug-symbol collection on Linux, macOS, and Windows: hf-xet diagnostic scripts.
Control
I first downloaded:
- repository:
openai-community/gpt2
- file:
model.safetensors
- Xet client:
hf-xet 1.5.1
- fresh cache and destination
That completed normally in the same runtime.
This does not prove that Xet was generally healthy everywhere, but it weakens several broad explanations:
- the Colab runtime could not use Xet at all;
- the installed client failed for every public file;
- the runtime had no usable network path;
- the destination filesystem was generally unwritable;
- Hugging Face large-file downloads were completely unavailable.
The control instead points toward something specific to the target object, its reconstruction plan, or a code/backend path triggered by that object.
Baseline: hf-xet 1.5.1
With the target file and default Xet settings, the download made rapid initial progress and then entered a no-progress state.
The exact numbers depend on which layer is measured:
- the Xet log had observed roughly 2.0 GB of transferred range data;
- the locally materialized destination lagged substantially behind the transferred amount;
- destination, cache, and logs then stopped changing;
- the process remained alive rather than returning a normal error;
- the watchdog terminated it after four minutes without observable progress.
This distinction matters because these quantities are not interchangeable:
| Quantity |
Meaning |
| CLI progress display |
User-facing reconstruction/download progress |
| Xet observed transferred bytes |
Data seen by the network/range-transfer layer |
| Destination logical size |
File length reported by the filesystem |
| Destination allocated/materialized bytes |
Data actually backed by local blocks |
| Xet cache size |
Cached reconstruction or transfer data |
Therefore, “it stops at 2 GB” should not automatically be interpreted as “the server cannot return byte 2,000,000,001.”
The logs instead suggest that a substantial amount of range traffic had already succeeded before the pipeline stopped completing useful local work.
Xet-disabled compatibility path
I then ran the same target with:
HF_HUB_DISABLE_XET=1
This branch passed:
2,569,011,200 bytes
before the test intentionally stopped it for crossing the configured 2.5 GB boundary.
That is the strongest discriminator from this matrix.
It means that, from the same Colab runtime:
- the repository was accessible;
- the file could be addressed;
- data beyond the reported stopping region could be retrieved;
- the local filesystem could write beyond 2 GB;
- the target was not universally unreadable after that point.
This does not prove the backing object is perfectly healthy. A different route can mask or bypass some storage/reconstruction problem. It does, however, make a fixed “all paths fail at this file offset” explanation difficult to maintain.
The compatibility run was not perfectly clean. It logged incomplete response bodies resembling:
peer closed connection without sending complete message body
The downloader resumed and continued past the threshold. Consequently:
- the non-Xet path appears more resilient for this case;
- it is not evidence that the network or HTTP serving path was completely error-free;
- passing 2.5 GB is a path-isolation result, not a full completion guarantee.
Sequential reconstruction writes
I tested:
HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY=1
This branch also entered a no-progress state.
The official documentation says this changes local reconstruction from parallel direct-address writes to sequential writes, primarily for slower spinning disks: Xet environment-variable reference.
Because sequential writing did not resolve the stall, the simplest version of the following hypothesis becomes less likely:
The failure is caused only by parallel random writes to the local destination.
It does not eliminate the broader reconstruction/write pipeline. The client can still:
- fetch multiple ranges;
- buffer reconstructed data;
- wait for ordering or completion conditions;
- encounter backpressure before a sequential write;
- wait on a missing or failed range.
So I would phrase this as:
Switching the final local write mode alone did not remove the failure.
Attempted one-range concurrency branch
I also set:
HF_XET_NUM_CONCURRENT_RANGE_GETS=1
However, I do not think this produced a clean “single Range request at a time” experiment.
Although the environment variable was present before the Hub client started, the runtime log initialized its adaptive-concurrency controller at a concurrency of four and later adjusted it upward. It did not visibly behave as a fixed concurrency of one.
The official Hub documentation describes HF_XET_NUM_CONCURRENT_RANGE_GETS as the number of concurrent byte-range terms downloaded per file and currently lists a default of 16: Hugging Face Hub environment variables.
Because the runtime log did not reflect the intended fixed value, the correct conclusion is not:
Concurrency one still fails.
The correct conclusion is:
This branch did not cleanly isolate single-range execution, so it is inconclusive.
That discrepancy may itself be useful to note in #896, especially if adaptive concurrency has superseded or interacts with this setting in the current client.
hf-xet 1.5.2rc0
The release-candidate branch changed the failure mode.
With hf-xet 1.5.2rc0, the download did not remain silently stuck indefinitely. It exited with an error of the following form:
RuntimeError: Task error:
File reconstruction error:
CAS Client Error:
Format error: I/O error: error decoding response body
The Xet log showed repeated retries for range requests and then an explicit abort similar to:
No more retries; aborting:
Format error: I/O error: error decoding response body
By that time, the transfer layer had observed roughly 2.2 GB.
I would not call this a fix. The file still did not pass the boundary through the Xet-aware path.
It is nevertheless diagnostically useful because it changes:
1.5.1: live process with no observable progress;1.5.2rc0: finite retries followed by an explicit response-body decoding failure.
One possible interpretation is that the release candidate surfaces a transfer failure that 1.5.1 can wait on indefinitely. Another is that the versions simply reach different failures under unstable network responses. Maintainer analysis is needed to distinguish those possibilities.
For #896, the most valuable comparison may be:
The stable client hangs, while the release candidate turns the same general run into a retryable/terminal response-body decoding error around the same transfer region.
That gives maintainers a more concrete error path to inspect than the progress bar alone.
Older-client branch
I attempted to test hf-xet 1.4.3, but that branch failed before the download started with an import compatibility error involving XetFileInfo.
The Hub client had been pinned independently, and that older hf-xet build did not provide the interface expected by the selected huggingface_hub version.
Therefore this branch provides no evidence about whether the underlying stall existed in 1.4.3.
A valid regression test would need compatible pairs of:
for each historical point.
I would not delay the current report for that matrix. The stable-versus-release-candidate comparison and Xet-disabled result are already more directly useful.
Why the approximately-2-GB region is notable
The stopping region is now visible in several independent observations:
- the original macOS reproduction;
- the RunPod reproduction;
- the earlier Colab reproduction;
- the new
hf-xet 1.5.1 transfer log;
- the
1.5.2rc0 failure after a similar amount of transferred range data.
The Xet log also printed an internal configuration value corresponding to a 2 GB download buffer.
That numerical proximity is worth preserving as a clue, but it is not enough to claim a 2 GB buffer-boundary bug.
The measurements refer to different things:
- user-visible reconstructed progress;
- bytes received by the transfer layer;
- bytes materialized locally;
- configured buffer capacity.
A deterministic reconstruction segment, one unavailable range, an ordering barrier, or a response-body failure could all produce numbers near a buffer capacity without the buffer implementation itself being defective.
A careful public formulation would be:
The repeated proximity to approximately 2 GB, including a logged 2 GB download-buffer configuration, is notable, but the available evidence does not identify the buffer boundary as the cause.
Current hypothesis map
The experiment changes the relative weight of the hypotheses as follows.
| Hypothesis |
Effect of the new results |
| macOS-specific issue |
Strongly weakened |
| RunPod-specific issue |
Strongly weakened |
| Colab-specific issue |
Weakened by macOS and RunPod reproduction |
| User account or token problem |
Strongly weakened; tests were public and unauthenticated |
| Local filesystem cannot exceed 2 GB |
Strongly weakened; Xet-disabled path passed 2.5 GB |
| File is universally unreadable after 2 GB |
Strongly weakened |
| Xet is broken for every public model |
Weakened by successful GPT-2 control |
| Parallel destination writes are the sole cause |
Weakened by sequential-write stall |
| Xet-aware reconstruction/fetch path is involved |
Strengthened |
| Incomplete HTTP/range responses contribute |
Strengthened by 1.5.2rc0 decoding error and fallback resumes |
| Target-specific reconstruction plan or range |
Still plausible |
| Xet client waits indefinitely after a failed task |
Plausible, not confirmed |
| Shared backend/CAS/CDN dependency |
Still plausible |
| Exact 2 GB buffer bug |
Interesting but unconfirmed |
The most defensible summary is:
The target can pass the reported boundary through the Xet-disabled compatibility path, while multiple Xet-aware branches either stop making progress or terminate after response-body decoding failures. This narrows the problem to the Xet-aware path or an interaction specific to that path, but does not yet distinguish client state management from a target-specific backend/range failure.
Relation to the direct AccessDenied reports
In the same time window, I also found a separate direct-download failure mode discussed in:
Access Denied trying to download any model
Repeated direct HTTP Range requests could alternate between:
- successful
206 responses through one terminal delivery route;
- XML
403 AccessDenied responses through another route.
That finding is relevant as evidence that file delivery was not uniformly healthy during the incident window.
However, I would not merge the two observations into one causal chain:
- the direct test exercised the browser/HTTP-compatible resolver path;
- the Xet-aware run requested reconstruction data and byte ranges through the Xet machinery;
- the full Xet stall did not emit the same bridge XML
403;
1.5.2rc0 instead exposed a response-body decoding error.
A safe formulation is:
The direct route-dependent 403 and the approximately-2-GB Xet stall occurred in the same broader period, but neither has yet been shown to cause the other.
Similar open reports
The exact deterministic behavior in #896 is fairly specific, but there are related open reports:
These reports support a broader “large Xet transfers can enter a no-progress state” failure class. They do not establish that every report shares #896’s exact trigger.
The distinguishing value of #896 is:
- one specific file;
- similar stopping region;
- multiple operating systems;
- official diagnostic bundle;
- now an independent control/fallback matrix.
What seems most useful to add to #896
The issue already contains the original reproduction and official diagnostic archive, so I would avoid burying it under many speculative tests.
A concise follow-up could preserve these points:
- A fresh Colab runtime reproduced the Xet-aware no-progress state.
- A public GPT-2 Xet control completed.
- Sequential reconstruction writes did not resolve the target stall.
- The intended one-range concurrency test was inconclusive because the adaptive-concurrency log did not remain at one.
hf-xet 1.5.2rc0 converted the silent stall into an explicit response-body decoding error after retries.HF_HUB_DISABLE_XET=1 passed 2.5 GB on the same target.- The historical
1.4.3 test was invalid because of a Hub/Xet interface mismatch.
- Full logs should have signed URLs, tokens, and authorization headers redacted.
If maintainers request one narrower follow-up, I would prioritize reproducing the 1.5.2rc0 error while preserving:
- failing terminal hostname;
- requested Range header;
- response status and headers;
- retry count;
- Xet task identifier;
- whether every retry targets the same byte range;
- whether a direct request for that same range succeeds.
That has a better chance of separating:
- an incomplete network response;
- one bad object/range;
- route instability;
- a decoder/client problem;
- retry exhaustion followed by incorrect task completion behavior.
Safe handling of diagnostics
The official Xet diagnostic scripts are intended to produce bundles that can be attached to GitHub issues. Before posting any manually collected material, it is still worth checking for:
Authorization headers;HF_TOKEN;- signed URL query parameters;
- cookies;
- private repository names or paths;
- usernames and home-directory paths where relevant.
For URLs, the hostname and path are usually sufficient for public diagnosis. Signed query parameters should be removed.
The existing diagnostic bundle plus this control/fallback comparison should give xet-core #896 a fairly concrete starting point.