Welcome to new things

[Technical] [Electronic work] [Gadget] [Game] memo writing

How to get directory listing in AWS S3 in JSON

In the following directory structure, how to get a list of directories directly under a bucket or "dir_A1/dir_B1" in JSON.

  • s3://bucket_1/

    • dir_A1/

      • dir_B1/

        • dir_C1/
        • dir_C2/
        • dir_C3/
    • dir_A2/
    • dir_A3/

When directly under the bucket

$ aws s3api list-object-v2 --bucket 'bucket_1' --prefix '' --delimiter '/' --output json
{
    "CommonPrefixes": [
        {
            "Prefix": "dir_A1/"
        },
        {
            "Prefix": "dir_A2/"
        },
        {
            "Prefix": "dir_A3/"
        }
    ]
}

Directly under a subdirectory

$ aws s3api list-object-v2 --bucket 'bucket_1' --prefix 'dir_A1/dir_B1/' --delimiter '/' --output json
{
    "CommonPrefixes": [
        {
            "Prefix": "dir_A1/dir_B1/dir_C1/"
        },
        {
            "Prefix": "dir_A1/dir_B1/dir_C2/"
        },
        {
            "Prefix": "dir_A1/dir_B1/dir_C3/"
        }
    ]
}

Specify the path of the directory to be examined in "prefix" (including the trailing "/") and set "/" in "delimiter".

Then, the list of path characters starting with "prefix" and ending with the next "delimiter" character is stored in "CommonPrefixes" and is used as the directory listing.

A similar list can be obtained with the aws s3 ls command, but the aws s3 command does not output JSON, so the aws s3api command was used.

Even when using the SDK, the same results can be obtained by specifying the same parameters in the corresponding command for "list-objects-v2".

Other

At first, I thought I could do it quickly with aws s3 ls ~, but it is surprisingly time-consuming.

In fact, there is no such thing as a directory. A bucket is a collection of "key-values" of keys (=paths) and contents (=files), and the keys are separated by "/" to make it look like a directory.

So, strictly speaking, it is not a directory listing, but a collection of prefixes, as you can see from "CommonPrefixes".

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com

www.ekwbtblog.com