Hello,
I was previously happily using the pytorchligning/pytorch_lightning docker image to run training, and I used s3:// urls as checkpoint directories and tensorboard logger paths. However, a couple of days ago this started failing with the error below.
Note that I do “pip install s3fs” on the docker to enable the s3 support. It would be probably good to have s3fs on the docker image itself.
Any advice how to resolve this issue? It seems to be some backward/forward compatibility issue.
Traceback (most recent call last):
2021-11-30 10:49:22.836 [545/train/control-545-start_train-2081 (pid 49012)] [57fe6c23-6c08-4928-8425-c7e54de68475] File "/usr/local/lib/python3.9/dist-packages/pytorch_lightning/loggers/tensorboard.py", line 229, in log_metrics
2021-11-30 10:49:22.836 [545/train/control-545-start_train-2081 (pid 49012)] [57fe6c23-6c08-4928-8425-c7e54de68475] self.experiment.add_scalar(k, v, step)
2021-11-30 10:49:22.836 [545/train/control-545-start_train-2081 (pid 49012)] [57fe6c23-6c08-4928-8425-c7e54de68475] File "/usr/local/lib/python3.9/dist-packages/pytorch_lightning/loggers/base.py", line 43, in experiment
2021-11-30 10:49:22.836 [545/train/control-545-start_train-2081 (pid 49012)] [57fe6c23-6c08-4928-8425-c7e54de68475] return get_experiment() or DummyExperiment()
2021-11-30 10:49:22.836 [545/train/control-545-start_train-2081 (pid 49012)] [57fe6c23-6c08-4928-8425-c7e54de68475] File "/usr/local/lib/python3.9/dist-packages/pytorch_lightning/utilities/distributed.py", line 49, in wrapped_fn
2021-11-30 10:49:22.837 [545/train/control-545-start_train-2081 (pid 49012)] [57fe6c23-6c08-4928-8425-c7e54de68475] return fn(args, kwargs)
2021-11-30 10:49:22.837 [545/train/control-545-start_train-2081 (pid 49012)] [57fe6c23-6c08-4928-8425-c7e54de68475] File "/usr/local/lib/python3.9/dist-packages/pytorch_lightning/loggers/base.py", line 41, in get_experiment
2021-11-30 10:49:22.837 [545/train/control-545-start_train-2081 (pid 49012)] [57fe6c23-6c08-4928-8425-c7e54de68475] return fn(self)
2021-11-30 10:49:22.837 [545/train/control-545-start_train-2081 (pid 49012)] [57fe6c23-6c08-4928-8425-c7e54de68475] File "/usr/local/lib/python3.9/dist-packages/pytorch_lightning/loggers/tensorboard.py", line 172, in experiment
2021-11-30 10:49:22.837 [545/train/control-545-start_train-2081 (pid 49012)] [57fe6c23-6c08-4928-8425-c7e54de68475] self._fs.makedirs(self.root_dir, exist_ok=True)
2021-11-30 10:49:22.837 [545/train/control-545-start_train-2081 (pid 49012)] [57fe6c23-6c08-4928-8425-c7e54de68475] File "/usr/local/lib/python3.9/dist-packages/fsspec/asyn.py", line 91, in wrapper
2021-11-30 10:49:22.837 [545/train/control-545-start_train-2081 (pid 49012)] [57fe6c23-6c08-4928-8425-c7e54de68475] return sync(self.loop, func, args, kwargs)
2021-11-30 10:49:22.837 [545/train/control-545-start_train-2081 (pid 49012)] [57fe6c23-6c08-4928-8425-c7e54de68475] File "/usr/local/lib/python3.9/dist-packages/fsspec/asyn.py", line 71, in sync
2021-11-30 10:49:22.837 [545/train/control-545-start_train-2081 (pid 49012)] [57fe6c23-6c08-4928-8425-c7e54de68475] raise return_result
2021-11-30 10:49:22.837 [545/train/control-545-start_train-2081 (pid 49012)] [57fe6c23-6c08-4928-8425-c7e54de68475] File "/usr/local/lib/python3.9/dist-packages/fsspec/asyn.py", line 25, in _runner
2021-11-30 10:49:22.837 [545/train/control-545-start_train-2081 (pid 49012)] [57fe6c23-6c08-4928-8425-c7e54de68475] result[0] = await coro
2021-11-30 10:49:22.837 [545/train/control-545-start_train-2081 (pid 49012)] [57fe6c23-6c08-4928-8425-c7e54de68475] File "/usr/local/lib/python3.9/dist-packages/s3fs/core.py", line 733, in _makedirs
2021-11-30 10:49:22.837 [545/train/control-545-start_train-2081 (pid 49012)] [57fe6c23-6c08-4928-8425-c7e54de68475] await self._mkdir(path, create_parents=True)
2021-11-30 10:49:22.837 [545/train/control-545-start_train-2081 (pid 49012)] [57fe6c23-6c08-4928-8425-c7e54de68475] File "/usr/local/lib/python3.9/dist-packages/s3fs/core.py", line 718, in _mkdir
2021-11-30 10:49:22.837 [545/train/control-545-start_train-2081 (pid 49012)] [57fe6c23-6c08-4928-8425-c7e54de68475] await self._call_s3("create_bucket", params)
2021-11-30 10:49:22.837 [545/train/control-545-start_train-2081 (pid 49012)] [57fe6c23-6c08-4928-8425-c7e54de68475] File "/usr/local/lib/python3.9/dist-packages/s3fs/core.py", line 241, in _call_s3
2021-11-30 10:49:22.837 [545/train/control-545-start_train-2081 (pid 49012)] [57fe6c23-6c08-4928-8425-c7e54de68475] await self.set_session()
2021-11-30 10:49:22.837 [545/train/control-545-start_train-2081 (pid 49012)] [57fe6c23-6c08-4928-8425-c7e54de68475] File "/usr/local/lib/python3.9/dist-packages/s3fs/core.py", line 409, in set_session
2021-11-30 10:49:22.837 [545/train/control-545-start_train-2081 (pid 49012)] [57fe6c23-6c08-4928-8425-c7e54de68475] self._s3 = await s3creator.__aenter__()
2021-11-30 10:49:22.837 [545/train/control-545-start_train-2081 (pid 49012)] [57fe6c23-6c08-4928-8425-c7e54de68475] File "/usr/local/lib/python3.9/dist-packages/aiobotocore/session.py", line 37, in __aenter__
2021-11-30 10:49:22.837 [545/train/control-545-start_train-2081 (pid 49012)] [57fe6c23-6c08-4928-8425-c7e54de68475] self._client = await self._coro
2021-11-30 10:49:22.837 [545/train/control-545-start_train-2081 (pid 49012)] [57fe6c23-6c08-4928-8425-c7e54de68475] File "/usr/local/lib/python3.9/dist-packages/aiobotocore/session.py", line 121, in _create_client
2021-11-30 10:49:22.837 [545/train/control-545-start_train-2081 (pid 49012)] [57fe6c23-6c08-4928-8425-c7e54de68475] client = await client_creator.create_client(
2021-11-30 10:49:22.837 [545/train/control-545-start_train-2081 (pid 49012)] [57fe6c23-6c08-4928-8425-c7e54de68475] File "/usr/local/lib/python3.9/dist-packages/aiobotocore/client.py", line 46, in create_client
2021-11-30 10:49:22.837 [545/train/control-545-start_train-2081 (pid 49012)] [57fe6c23-6c08-4928-8425-c7e54de68475] self._register_lazy_block_unknown_fips_pseudo_regions(service_client)
2021-11-30 10:49:22.837 [545/train/control-545-start_train-2081 (pid 49012)] [57fe6c23-6c08-4928-8425-c7e54de68475] AttributeError: 'AioClientCreator' object has no attribute '_register_lazy_block_unknown_fips_pseudo_regions'```