I usually use Google BigQuery only for simple aggregate SQL, but there are many other useful functions besides aggregate SQL. However, since I don't use it often and when I do try to use it, I start by looking for a manual, so I'll put tog…

I was working on a program related to GCP and wanted to store some data. It was a one-off piece of data that was not large enough to set up a database, nor was it likely to write a join query between the data. To begin with, setting up a d…

I usually use Google BigQuery only for simple aggregate SQL, but there are many other useful functions besides aggregate SQL. However, since I don't use it often and when I do try to use it, I start by looking for a manual, so I'm going to…

I usually use Google BigQuery only for simple aggregate SQL, but there are many other useful functions besides aggregate SQL. However, since I don't use it often and when I do try to use it, I start by looking for a manual, so I'll put tog…

I usually use Google BigQuery only for simple aggregate SQL, but there are many other useful functions besides aggregate SQL. However, since I don't use it often and when I do try to use it, I start by looking for a manual, so I'll put tog…

I usually use Google BigQuery only for simple aggregate SQL, but there are many other useful functions besides aggregate SQL. However, since I don't use it often and when I do try to use it, I start by looking for a manual, so I'll put tog…

Normally, GCP operations can be done via command line tools or client libraries, so we never called the REST API directly. I'm trying out a service called GCP Batch right now, but it's still in preview stage and the command line tools and …

Since Google BigQuery is pay-as-you-go and relatively inexpensive, I had been using it without much concern about the fees. Big data analysis was a breakthrough in that matters that until now could only be guessed at could actually be expr…

I had the opportunity to take the Google Professional Data Engineer certification exam and I passed! There is not much information about Google Professional Data Engineer certification, so I will try to write a success story for those who …

I use multiple PCs and have Visual Studio Code (VSCode) installed on each PC. I often think that if VSCode could be used on a browser, it would be easier to use the same environment anytime without the need for setup. GitHub Codespaces is …

I had hoped to be able to create a quick SPA site on the spur of the moment, but I had given up on React because it is too big a job to create even a small site. However, I recently started touching Recoil for React, and with Recoil, React…

Goroutine Leak The Go language is a fascinating language because it is so easy to create goroutines (like threads). However, since the goroutine is executed separately from the main flow, it was unnoticed when the goroutine process stopped…

This is a way to notify external parties such as Slack or email when an error occurs in a program in GCP's Kubernetes (GKE). You do not have to implement your own error detection mechanism, only GCP settings can do this. How it works When …

We have added a new Kubernetes cluster and configured it to be switched with the kubectl command. Once you set it up, you don't have to tweak it, and if you don't, you'll forget how to do it, so I'll write down how to set it up for myself.…

We added a new project to Google Cloud Platform and set up the Google Cloud SDK (gcloud CLI) to switch projects. Once you set it up, you don't have to tweak it, and if you don't, you'll forget how to do it, so I'll write down how to set it…

Periodically, AWS S3 files are copied to Google Cloud Storage. There are many ways to copy, but here we use the gsutil command for the Google Cloud SDK, The Google Cloud SDK uses a Docker image of Alpine Linux. However, at some point, I st…

When loading Google Cloud Storage files into Google BigQuery, multiple files can be loaded at once. However, I got stuck trying to do that with Google BigQuery's Node.js SDK, so here are my notes on that. Overview and Issues Load using tab…

In Google Cloud Storage, I tried to get a list of files in a subdirectory and process those files. However, when I looked at the error message that the file was wrong, I found that the list of files retrieved included a subdirectory. Reaso…

We previously used the GCP Pyhton library to get a list of Google Cloud Storage subdirectories, and this is the Node.js version of that. Python version is here www.ekwbtblog.com procedure The GCP Node.js SDK documentation is kind enough to…

I received an email from Google Cloud Platform. The Node.js version of the function I originally created for Gooble Cloud Functions was version 6, and this time the version 6 service was ending, so the email was a request for an upgrade. I…

How to access AWS S3 from Spark (Google Dataproc). procedure Spark Configuration The following Spark and Haddop settings will allow you to read and write AWS S3 files from Spark. Load the following AWS-related jar files into Spark aws-java…

How to access Microsoft SQL Server (Azure Database) from Spark (Google Dataproc). procedure Spark Configuration The following Spark settings will allow you to read and write SQL Server data from Spark. Download the MS SQL Server JDBC jar f…

How to access MySQL from Spark (Google Dataproc). Since it is accessed using JDBC, it can be applied to other RDBs such as PostgreSQL. procedure Spark Configuration The following Spark settings will allow you to read and write MySQL data f…

BigQuery can handle huge amounts of data, but you don't have to worry about the infrastructure at all (really, at all), and it's fast and cheap, It is tempting to put all your data into BigQuery and process it all with BigQuery. That's why…

I wanted to get a list of Google Cloud Storage subdirectories using the GCP Python library, I got stuck, so here's a note on how and why. Libraries in other languages are API wrappers as well as Python libraries, so I think they can be app…

It was convenient to use "cert-manager" + DNS authentication to automatically update Let's Encrypt on Kubernetes (GKE) without processing on the service side. I wrote Introductory note before, but the procedure had changed, so I rewrote it…

When aggregating data that jumps around by date, you may want a calendar table. You can create a calendar table in Google BigQuery with the following GENERATE_DATE_ARRAY()" creates a calendar array, and "UNNEST()" expands the array to rows…

PS Since the procedure had changed, we have rewritten it in a separate article, along with how to obtain a wildcard certificate. www.ekwbtblog.com Postscript ends here (original article below) It was convenient to use "cert-manager" + DNS …

This is a summary of how to communicate over HTTPS with Kubernetes (GKE) on Google Cloud Platform. Introduction. To allow Kubernetes to communicate with the outside world Method by Service Methods by Ingress There are two ways to do this. …

Google Cloud Platform (GCP) has two types of accounts: user accounts and service accounts. A user account is a Google account, familiarly known as Gmail, that can belong to multiple arbitrary projects. A service account, on the other hand,…