| 2026-02-27 13:02:22,464 INFO StreamThr :1510588 [internal.py:wandb_internal():85] W&B internal server running at pid: 1510588, started at: 2026-02-27 13:02:22.460261 |
| 2026-02-27 13:02:22,465 DEBUG HandlerThread:1510588 [handler.py:handle_request():158] handle_request: status |
| 2026-02-27 13:02:23,404 INFO WriterThread:1510588 [datastore.py:open_for_write():87] open: /proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/bimamba_stage_a/wandb/run-20260227_130222-lxvmjzq8/run-lxvmjzq8.wandb |
| 2026-02-27 13:02:23,407 DEBUG SenderThread:1510588 [sender.py:send():391] send: header |
| 2026-02-27 13:02:23,489 DEBUG SenderThread:1510588 [sender.py:send():391] send: run |
| 2026-02-27 13:02:23,507 INFO SenderThread:1510588 [sender.py:_setup_resume():761] checking resume status for turlin/worldmem/lxvmjzq8 |
| 2026-02-27 13:02:24,031 ERROR SenderThread:1510588 [internal_api.py:execute():349] 403 response executing GraphQL. |
| 2026-02-27 13:02:24,031 ERROR SenderThread:1510588 [internal_api.py:execute():350] {"errors":[{"message":"permission denied","path":["upsertBucket"],"extensions":{"code":"PERMISSION_ERROR"}}],"data":{"upsertBucket":null}} |
| 2026-02-27 13:02:24,032 ERROR SenderThread:1510588 [sender.py:send_run():1021] It appears that you do not have permission to access the requested resource. Please reach out to the project owner to grant you access. If you have the correct permissions, verify that there are no issues with your networking setup.(Error 403: Forbidden) |
| Traceback (most recent call last): |
| File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/sdk/lib/retry.py", line 131, in __call__ |
| result = self._call_fn(*args, **kwargs) |
| File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/sdk/internal/internal_api.py", line 345, in execute |
| return self.client.execute(*args, **kwargs) # type: ignore |
| File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/vendor/gql-0.2.0/wandb_gql/client.py", line 52, in execute |
| result = self._get_result(document, *args, **kwargs) |
| File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/vendor/gql-0.2.0/wandb_gql/client.py", line 60, in _get_result |
| return self.transport.execute(document, *args, **kwargs) |
| File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/sdk/lib/gql_request.py", line 59, in execute |
| request.raise_for_status() |
| File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/requests/models.py", line 1026, in raise_for_status |
| raise HTTPError(http_error_msg, response=self) |
| requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://api.wandb.ai/graphql |
|
|
| During handling of the above exception, another exception occurred: |
|
|
| Traceback (most recent call last): |
| File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/sdk/internal/sender.py", line 1019, in send_run |
| server_run = self._init_run(run, config_value_dict) |
| File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/sdk/internal/sender.py", line 1091, in _init_run |
| server_run, inserted, server_messages = self._api.upsert_run( |
| File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/apis/normalize.py", line 73, in wrapper |
| raise err |
| File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/apis/normalize.py", line 41, in wrapper |
| return func(*args, **kwargs) |
| File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/sdk/internal/internal_api.py", line 2224, in upsert_run |
| response = self.gql( |
| File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/sdk/internal/internal_api.py", line 317, in gql |
| ret = self._retry_gql( |
| File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/sdk/lib/retry.py", line 147, in __call__ |
| retry_timedelta_triggered = check_retry_fn(e) |
| File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/util.py", line 968, in check_retry_fn |
| return fallback_retry_fn(e) |
| File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/util.py", line 910, in no_retry_auth |
| raise CommError( |
| wandb.errors.CommError: It appears that you do not have permission to access the requested resource. Please reach out to the project owner to grant you access. If you have the correct permissions, verify that there are no issues with your networking setup.(Error 403: Forbidden) |
| 2026-02-27 13:02:24,190 DEBUG HandlerThread:1510588 [handler.py:handle_request():158] handle_request: shutdown |
| 2026-02-27 13:02:24,190 INFO HandlerThread:1510588 [handler.py:finish():884] shutting down handler |
| 2026-02-27 13:02:24,489 INFO WriterThread:1510588 [datastore.py:close():296] close: /proj/cvl/users/x_fahkh2/WorldMem_Repro/checkpoints/bimamba_stage_a/wandb/run-20260227_130222-lxvmjzq8/run-lxvmjzq8.wandb |
| 2026-02-27 13:02:25,058 INFO SenderThread:1510588 [sender.py:finish():1631] shutting down sender |
|
|
