2026-02-11 07:41:02,301 INFO    StreamThr :1296208 [internal.py:wandb_internal():85] W&B internal server running at pid: 1296208, started at: 2026-02-11 07:41:02.291990
2026-02-11 07:41:02,302 DEBUG   HandlerThread:1296208 [handler.py:handle_request():158] handle_request: status
2026-02-11 07:41:02,389 INFO    WriterThread:1296208 [datastore.py:open_for_write():87] open: /proj/cvl/users/x_fahkh2/WorldModels/WorldMem_Repro/reproduce_official_set/wandb/run-20260211_074102-ot7jqmgn/run-ot7jqmgn.wandb
2026-02-11 07:41:02,395 DEBUG   SenderThread:1296208 [sender.py:send():391] send: header
2026-02-11 07:41:02,448 DEBUG   SenderThread:1296208 [sender.py:send():391] send: run
2026-02-11 07:41:02,465 INFO    SenderThread:1296208 [sender.py:_setup_resume():761] checking resume status for turlin/worldmem/ot7jqmgn
2026-02-11 07:41:03,000 ERROR   SenderThread:1296208 [internal_api.py:execute():349] 403 response executing GraphQL.
2026-02-11 07:41:03,000 ERROR   SenderThread:1296208 [internal_api.py:execute():350] {"errors":[{"message":"permission denied","path":["upsertBucket"],"extensions":{"code":"PERMISSION_ERROR"}}],"data":{"upsertBucket":null}}
2026-02-11 07:41:03,000 ERROR   SenderThread:1296208 [sender.py:send_run():1021] It appears that you do not have permission to access the requested resource. Please reach out to the project owner to grant you access. If you have the correct permissions, verify that there are no issues with your networking setup.(Error 403: Forbidden)
Traceback (most recent call last):
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/sdk/lib/retry.py", line 131, in __call__
    result = self._call_fn(*args, **kwargs)
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/sdk/internal/internal_api.py", line 345, in execute
    return self.client.execute(*args, **kwargs)  # type: ignore
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/vendor/gql-0.2.0/wandb_gql/client.py", line 52, in execute
    result = self._get_result(document, *args, **kwargs)
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/vendor/gql-0.2.0/wandb_gql/client.py", line 60, in _get_result
    return self.transport.execute(document, *args, **kwargs)
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/sdk/lib/gql_request.py", line 59, in execute
    request.raise_for_status()
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/requests/models.py", line 1026, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://api.wandb.ai/graphql

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/sdk/internal/sender.py", line 1019, in send_run
    server_run = self._init_run(run, config_value_dict)
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/sdk/internal/sender.py", line 1091, in _init_run
    server_run, inserted, server_messages = self._api.upsert_run(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/apis/normalize.py", line 73, in wrapper
    raise err
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/apis/normalize.py", line 41, in wrapper
    return func(*args, **kwargs)
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/sdk/internal/internal_api.py", line 2224, in upsert_run
    response = self.gql(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/sdk/internal/internal_api.py", line 317, in gql
    ret = self._retry_gql(
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/sdk/lib/retry.py", line 147, in __call__
    retry_timedelta_triggered = check_retry_fn(e)
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/util.py", line 968, in check_retry_fn
    return fallback_retry_fn(e)
  File "/proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/wandb/util.py", line 910, in no_retry_auth
    raise CommError(
wandb.errors.CommError: It appears that you do not have permission to access the requested resource. Please reach out to the project owner to grant you access. If you have the correct permissions, verify that there are no issues with your networking setup.(Error 403: Forbidden)
2026-02-11 07:41:03,358 DEBUG   HandlerThread:1296208 [handler.py:handle_request():158] handle_request: shutdown
2026-02-11 07:41:03,358 INFO    HandlerThread:1296208 [handler.py:finish():884] shutting down handler
2026-02-11 07:41:03,448 INFO    WriterThread:1296208 [datastore.py:close():296] close: /proj/cvl/users/x_fahkh2/WorldModels/WorldMem_Repro/reproduce_official_set/wandb/run-20260211_074102-ot7jqmgn/run-ot7jqmgn.wandb
2026-02-11 07:41:04,016 INFO    SenderThread:1296208 [sender.py:finish():1631] shutting down sender