How to access AWS S3 from Spark (Google Dataproc). procedure Spark Configuration The following Spark and Haddop settings will allow you to read and write AWS S3 files from Spark. Load the following AWS-related jar files into Spark aws-java…

How to access Microsoft SQL Server (Azure Database) from Spark (Google Dataproc). procedure Spark Configuration The following Spark settings will allow you to read and write SQL Server data from Spark. Download the MS SQL Server JDBC jar f…

How to access MySQL from Spark (Google Dataproc). Since it is accessed using JDBC, it can be applied to other RDBs such as PostgreSQL. procedure Spark Configuration The following Spark settings will allow you to read and write MySQL data f…

BigQuery can handle huge amounts of data, but you don't have to worry about the infrastructure at all (really, at all), and it's fast and cheap, It is tempting to put all your data into BigQuery and process it all with BigQuery. That's why…

I had a chance to do AES encryption/decryption in Node.js, so here are my notes. The encryption algorithm "AES-256-CBC" was used. About AES CBC encryption in a nutshell AES CBC is an encryption algorithm that encrypts binary data of arbitr…

I wanted to get a list of Google Cloud Storage subdirectories using the GCP Python library, I got stuck, so here's a note on how and why. Libraries in other languages are API wrappers as well as Python libraries, so I think they can be app…

When I saw the news of the Raspberry Pi4 announcement, I pulled out my Raspberry Pi and touched it for the first time in a while. I had touched the Raspberry Pi a little before, but gradually stopped using it because I was too lazy to move…