The Go language is often programmed by combining simple functions, and error checking occurs for each function, so the code in general tends to be long.
Even when I want to do something small, I need to write code accordingly, but it is tedious to write from scratch every time, so I would like to write down the processes I often do for my own copy and paste.
Here I have included my own notes on how to read and write using files (IO) in the Go language.
Introduction.
About io
In Go language, input/output is abstracted as io package, and read/write programs are written using io package.
The file has io.Reader and io.Writer interfaces, so you can use the read/write codes described using io without modification.
Not only files, but any other input/output can be used as long as it has the io.Reader and io.Writer interfaces.
About byte
All data exchange in io is done through the byte slice.
For example, when reading some data in io, the original data is in a slice of byte, that is, []byte.
Then, a portion of the data is acquired by io.Reader.Read(), and the acquired data is also a slice of byte.
The data to be written in io is written in io.Writer.Write(), and the data to be written is also a slice of byte.
File loading (the lowest level loading method)
Basic Flow
- Open the file and get
File Filehasio.Readerinterfaceio.Readerhas the methodRead()Read()writes data into the byte slice passed as argument and returns the number of bytes written
Termination Notes
- If a read terminates at a certain read, the number of reads may be greater than or equal to zero and
err==EOF(err!=nil). - Termination is
err==EOF - When the number of bytes read is zero, it may be the end of the line, but does not necessarily indicate the end of the line
So, the loading process requires the following
- Before checking for errors, check the number of reads and process reads if it is greater than zero.
- Termination check is performed by
err==EOF
f, err := os.Open("read_data.txt") // Open file as read defer f.Close() // Close opened files if err != nil { return err } const bufferSize = 256 // Read buffer size var content []byte buffer := make([]byte, bufferSize) // read buffer for { n, err := f.Read(buffer) if 0 < n { // read process content = append(content, buffer...) } if err == io.EOF { break // terminal } if err != nil { return nil, err // error } } fmt.Println(string(content))
File read (write all together)
When io.Read() is used, the contents are read one by one, but when ioutil.readAll() is used, all the data that io.Reader has is read at once.
f, err := os.Open("read_data.txt") defer f.Close() if err != nil { return err } content, err := ioutil.ReadAll(f) // It reads everything. if err != nil { return err } fmt.Println(string(content))
File loading (all together from file open)
If the target is a file, ioutil.ReadFile() will do everything from opening the file to reading and closing the file.
content, err := ioutil.ReadFile("read_data.txt") if err != nil { return err } fmt.Println(string(content))
This is the easiest way.
file writing
Write in io.Writer.Write().
f, err := os.Create("write_data.txt") defer f.Close() if err != nil { return err } content := "TEST\ntest\nテスト\nてすと" _, err = f.Write([]byte(content)) if err != nil { return err }
File write (buffer)
When the amount of data to be written is small, the above does not cause any problem, but when a large amount of data is written, it sometimes fails with an error.
In such cases, it is necessary to separate the data and write them one at a time, but bufio makes the task of separating and writing good.
If the write size is not clear, it is safer to use bufio instead of io.Writer.Write() directly.
treatment
The bufio creates an io.Writer that wraps the existing io.Writer and writes to that io.Writer, dividing the write into buffer sizes,
It will write to the existing io.Writer.
When the buffer size is reached, the accumulated data is written, so when the end of the data to be written is reached, Writer.Flash() must be used to write the remaining data that has accumulated in the buffer.
f, err := os.Create("write_data.txt") defer f.Close() if err != nil { return err } fw := bufio.NewWriter(f) content := "TEST\ntest\nテスト\nてすと" _, err = fw.Write([]byte(content)) if err != nil { return err } err = fw.Flush() if err != nil { return err }
File writing (all together from file open)
If the target is a file, ioutil.WriteFile() will do everything from opening the file to writing and closing the file.
content := "TEST\ntest\nテスト\nてすと" err := ioutil.WriteFile("write_data.txt", []byte(content), 0644) if err != nil { return err }
This is the easiest way.
Read from file one line at a time
The amount of data read from io.Reader at one time is indefinite. However, we may want to read one line of data at a time.
The aforementioned bufio has a function to retrieve data from io.Reader one line at a time.
Create Scanner from io.Reader.
The Scanner is used to obtain one line with Scan(), and when it is obtained, it becomes true, and the data can be brought in with Text().
If it terminates or an error occurs, Scan() becomes false, and the error is confirmed by Err().
f, err := os.Open("read_data.txt") defer f.Close() if err != nil { return err } fr := bufio.NewScanner(f) for fr.Scan() { fmt.Println(fr.Text()) } if err := fr.Err(); err != nil { return err }
Create io in memory
Use bytes.Buffer to set the data input/output destination to memory.
The bytes.Buffer has a []byte memory area for storing data and has io.Reader and io.Writer interfaces,
Data can be read and written to memory via the io.Reader and io.Writer interfaces.
Naturally, data can be retrieved with io.Reader, or all data can be directly retrieved with Bytes().
var b bytes.Buffer b.Write([]byte("TEST")) // Writing with io.Writer content, err := ioutil.ReadAll(&b) // Read in io.Reader if err != nil { return err } fmt.Println(string(content)) // TEST fmt.Println(string(b.Bytes())) // TEST Get data directly
Also, bytes.NewBuffer() can be used to create bytes.Buffer and initialize the data at the same time.
b := bytes.NewBuffer([]byte("TEST")) content, err := ioutil.ReadAll(b) // io.Reader if err != nil { return err } fmt.Println(string(content)) // TEST fmt.Println(string(b.Bytes())) // TEST
Points to note
The bytes.Buffer interface reads and writes memory data, while the io.Reader and io.Writer interfaces read and write memory data from and to the bytes.Buffer interface.pointerThe following is defined for the
So, to pass bytes.Buffer to io, you need to pass a pointer to bytes.Buffer.
This is why var b bytes.Buffer is passed as &b because it is an object, while b := bytes.NewBuffer() is passed as b because it is a pointer.
Direct connection between io.Reaer and io.Writer (stream)
Sometimes you want to read data from io.Reader and write it directly to io.Writer.
In such a case, outputting data after receiving all the data would be inefficient because it would require memory space to store all the data.
Therefore, if you read a part of the data with Read() and write it with Write() until io.Reader becomes EOF, you only need the memory for the data read by Read(),
This is called stream processing. This is what is called stream processing.
There is no need to write stream processing code. io.Copy() will do the work for you.
Open file and save as
r, err := Open("read_data.txt") if err != nil{ return err } defer r.Close() w, err := Create("write_data.txt") if err != nil{ return err } defer w.Close() // It reads from r and writes to w until the data in r is the last. _, err = io.Copy(w, r) if err != nil{ return err }
Direct connection between io.Writer and io.Reader (pipe)
To write data to io.Write and pass the written data to a function that takes io.Reader as an argument, use io.Pipe().
I was going to write an example, but I don't have many situations where I would use it, so I will just note the URL of the relevant information for reference in case I need it.
Impressions, etc.
If you have the target data, and you use io.Reader and io.Writer to create the data, you can use io.Reader and io.Writer to create the data.Partial.It is fitting to imagine reading, writing, and streaming.
Wrappers io.Reader and io.Writer for io.Reader and io.Writer are sometimes created and used for data manipulation and conversion, which can also be considered a type of stream.
If you resist the urge to read all the data once and then do the next process, and pass the data as io.Reader and io.Writer for stream processing, you can run many parallel processes, so I think you should actively use io.Reader and io.Writer as they are in Go language. I think it is better to use io.Reader and io.Writer as they are.
For example, zipping a file is handled by a stream as follows.
fr, err := os.Open("from.txt") if err != nil { return err } defer fr.Close() fw, err := os.Create("to.zip") if err != nil { return err } defer fw.Close() z := zip.NewWriter(fw) if err != nil { return err } defer z.Close() fz, err := z.Create("from.txt") if err != nil { return err } _, err = io.Copy(fz, fr) if err != nil { return err }
If written in this way, resource consumption is low, so if you want to process many files, you can execute a goroutine for each file, allowing many processes to be executed in parallel.
