Periodically, AWS S3 files are copied to Google Cloud Storage. There are many ways to copy, but here we use the gsutil command for the Google Cloud SDK, The Google Cloud SDK uses a Docker image of Alpine Linux. However, at some point, I st…

In Google Cloud Storage, I tried to get a list of files in a subdirectory and process those files. However, when I looked at the error message that the file was wrong, I found that the list of files retrieved included a subdirectory. Reaso…

We previously used the GCP Pyhton library to get a list of Google Cloud Storage subdirectories, and this is the Node.js version of that. Python version is here www.ekwbtblog.com procedure The GCP Node.js SDK documentation is kind enough to…

We originally used AWS Redshift, but it was quite expensive, so we switched to Google BigQuery. After that, Redshift Spectrum came out, and I was wondering if it might be better than BigQuery, so I touched it and here are my impressions. A…

There are times when you want to retrieve S3 files from one account to another. In such cases, it is quicker to create a dedicated user who can download the S3 file, but this can also be done by granting access to the destination S3 to the…

In the following directory structure, how to get a list of directories directly under a bucket or "dir_A1/dir_B1" in JSON. s3://bucket_1/ dir_A1/ dir_B1/ dir_C1/ dir_C2/ dir_C3/ dir_A2/ dir_A3/ When directly under the bucket $ aws s3api li…

I was building a single page application and did not want to publish it externally, but wanted to put it on a non-local server to test delivery, so here are my notes on building it. Elements Since it was a static distribution and S3 seemed…