| 2026-02-11 07:41:02,301 INFO StreamThr :1296208 [internal.py:wandb_internal():85] W&B internal server running at pid: 1296208, started at: 2026-02-11 07:41:02.291990 |
| 2026-02-11 07:41:02,302 DEBUG HandlerThread:1296208 [handler.py:handle_request():158] handle_request: status |
| 2026-02-11 07:41:02,389 INFO WriterThread:1296208 [datastore.py:open_for_write():87] open: /proj/cvl/users/x_fahkh2/WorldModels/WorldMem_Repro/reproduce_official_set/wandb/run-20260211_074102-ot7jqmgn/run-ot7jqmgn.wandb |
| 2026-02-11 07:41:02,395 DEBUG SenderThread:1296208 [sender.py:send():391] send: header |
| 2026-02-11 07:41:02,448 DEBUG SenderThread:1296208 [sender.py:send():391] send: run |
| 2026-02-11 07:41:02,465 INFO SenderThread:1296208 [sender.py:_setup_resume():761] checking resume status for turlin/worldmem/ot7jqmgn |
| 2026-02-11 07:41:03,000 ERROR SenderThread:1296208 [internal_api.py:execute():349] 403 response executing GraphQL. |
| 2026-02-11 07:41:03,000 ERROR SenderThread:1296208 [internal_api.py:execute():350] {"errors":[{"message":"permission denied","path":["upsertBucket"],"extensions":{"code":"PERMISSION_ERROR"}}],"data":{"upsertBucket":null}} |
| 2026-02-11 07:41:03,000 ERROR SenderThread:1296208 [sender.py:send_run():1021] It appears that you do not have permission to access the requested resource. Please reach out to the project owner to grant you access. If you have the correct permissions, verify that there are no issues with your networking setup.(Error 403: Forbidden) |
| Traceback (most recent call last): |
| File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/sdk/lib/retry.py", line 131, in __call__ |
| result = self._call_fn(*args, **kwargs) |
| File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/sdk/internal/internal_api.py", line 345, in execute |
| return self.client.execute(*args, **kwargs) # type: ignore |
| File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/vendor/gql-0.2.0/wandb_gql/client.py", line 52, in execute |
| result = self._get_result(document, *args, **kwargs) |
| File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/vendor/gql-0.2.0/wandb_gql/client.py", line 60, in _get_result |
| return self.transport.execute(document, *args, **kwargs) |
| File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/sdk/lib/gql_request.py", line 59, in execute |
| request.raise_for_status() |
| File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/requests/models.py", line 1026, in raise_for_status |
| raise HTTPError(http_error_msg, response=self) |
| requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://api.wandb.ai/graphql |
|
|
| During handling of the above exception, another exception occurred: |
|
|
| Traceback (most recent call last): |
| File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/sdk/internal/sender.py", line 1019, in send_run |
| server_run = self._init_run(run, config_value_dict) |
| File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/sdk/internal/sender.py", line 1091, in _init_run |
| server_run, inserted, server_messages = self._api.upsert_run( |
| File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/apis/normalize.py", line 73, in wrapper |
| raise err |
| File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/apis/normalize.py", line 41, in wrapper |
| return func(*args, **kwargs) |
| File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/sdk/internal/internal_api.py", line 2224, in upsert_run |
| response = self.gql( |
| File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/sdk/internal/internal_api.py", line 317, in gql |
| ret = self._retry_gql( |
| File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/sdk/lib/retry.py", line 147, in __call__ |
| retry_timedelta_triggered = check_retry_fn(e) |
| File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/util.py", line 968, in check_retry_fn |
| return fallback_retry_fn(e) |
| File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/util.py", line 910, in no_retry_auth |
| raise CommError( |
| wandb.errors.CommError: It appears that you do not have permission to access the requested resource. Please reach out to the project owner to grant you access. If you have the correct permissions, verify that there are no issues with your networking setup.(Error 403: Forbidden) |
| 2026-02-11 07:41:03,358 DEBUG HandlerThread:1296208 [handler.py:handle_request():158] handle_request: shutdown |
| 2026-02-11 07:41:03,358 INFO HandlerThread:1296208 [handler.py:finish():884] shutting down handler |
| 2026-02-11 07:41:03,448 INFO WriterThread:1296208 [datastore.py:close():296] close: /proj/cvl/users/x_fahkh2/WorldModels/WorldMem_Repro/reproduce_official_set/wandb/run-20260211_074102-ot7jqmgn/run-ot7jqmgn.wandb |
| 2026-02-11 07:41:04,016 INFO SenderThread:1296208 [sender.py:finish():1631] shutting down sender |
|
|
