Since Spark is a Python program, it can be written quite freely. However, since I always have a general idea of what I need to do, and knowing various ways of writing Spark makes it harder to remember, I will summarize my personal frequent…

How to access AWS S3 from Spark (Google Dataproc). procedure Spark Configuration The following Spark and Haddop settings will allow you to read and write AWS S3 files from Spark. Load the following AWS-related jar files into Spark aws-java…

How to access Microsoft SQL Server (Azure Database) from Spark (Google Dataproc). procedure Spark Configuration The following Spark settings will allow you to read and write SQL Server data from Spark. Download the MS SQL Server JDBC jar f…

How to access MySQL from Spark (Google Dataproc). Since it is accessed using JDBC, it can be applied to other RDBs such as PostgreSQL. procedure Spark Configuration The following Spark settings will allow you to read and write MySQL data f…

BigQuery can handle huge amounts of data, but you don't have to worry about the infrastructure at all (really, at all), and it's fast and cheap, It is tempting to put all your data into BigQuery and process it all with BigQuery. That's why…