Go File I/O — Reading and Writing Files with os, bufio, ioutil, and io Package

In this tutorial, you will learn about Go File I/O. We cover key concepts, practical examples, and best practices to help you master this topic.

Go file I/O uses os.File for low-level operations, bufio for buffered reading/writing, io package for streams, and os.ReadFile/WriteFile for simplicity.

What You'll Learn

• Reading and writing files
• Buffered I/O with bufio
• Directory and file operations
• File encoding and formats

Why It Matters

File I/O is fundamental. Docker reads Dockerfiles and images. Kubernetes reads config files. DodaZIP reads and writes compressed archives.

Real-World Use

Log file processing, configuration file Parsing, data export/import, file format conversion.

flowchart LR
    A["File I/O"] --> B["os.ReadFile"]
    A --> C["bufio.Scanner"]
    A --> D["os.File"]
    A --> E["Directories"]
    A:::current --> B
    style A fill:#2563eb,stroke:#2563eb,color:#fff
    style B fill:#dbeafe,stroke:#2563eb,color:#1e40af
    style C fill:#dbeafe,stroke:#2563eb,color:#1e40af
    style D fill:#dbeafe,stroke:#2563eb,color:#1e40af
    style E fill:#f1f5f9,stroke:#94a3b8,color:#64748b

Reading Entire File

func main() {
    content, err := os.ReadFile("data.txt")
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(string(content))
}

Writing Entire File

func main() {
    data := []byte("Hello, File!\nLine 2\n")
    err := os.WriteFile("output.txt", data, 0644)
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println("File written")
}

Buffered Reading

func main() {
    file, err := os.Open("data.txt")
    if err != nil {
        log.Fatal(err)
    }
    defer file.Close()

    scanner := bufio.NewScanner(file)
    for scanner.Scan() {
        fmt.Println(scanner.Text())
    }

    if err := scanner.Err(); err != nil {
        log.Fatal(err)
    }
}

Buffered Writing

func main() {
    file, err := os.Create("output.txt")
    if err != nil {
        log.Fatal(err)
    }
    defer file.Close()

    writer := bufio.NewWriter(file)
    for i := 0; i < 5; i++ {
        fmt.Fprintf(writer, "Line %d\n", i+1)
    }
    writer.Flush()
}

File Operations

func main() {
    // Copy file
    src, _ := os.Open("source.txt")
    dst, _ := os.Create("dest.txt")
    io.Copy(dst, src)
    src.Close()
    dst.Close()

    // File info
    info, _ := os.Stat("file.txt")
    fmt.Println("Size:", info.Size())
    fmt.Println("ModTime:", info.ModTime())

    // Rename/Move
    os.Rename("old.txt", "new.txt")

    // Delete
    os.Remove("temp.txt")
}

Directory Operations

func main() {
    // Create directory
    os.Mkdir("mydir", 0755)
    os.MkdirAll("a/b/c", 0755)

    // Read directory
    entries, _ := os.ReadDir(".")
    for _, entry := range entries {
        fmt.Println(entry.Name(), entry.IsDir())
    }

    // Walk recursively
    filepath.Walk(".", func(path string, info os.FileInfo, err error) error {
        if err != nil { return err }
        fmt.Println(path)
        return nil
    })

    // Remove recursively
    os.RemoveAll("mydir")
}

Common Mistakes

1. Forgetting to Close

file, _ := os.Open("file.txt")
// defer file.Close()  // Missing! File handle leaks

2. Not Checking Errors

content, _ := os.ReadFile("config.yaml")
// Unmarshal will fail silently or panic on nil

3. Wrong File Permissions

os.WriteFile("secret.txt", data, 0777)  // Too permissive!
// Use 0644 for files, 0755 for executables

4. Ignoring scanner.Err()

scanner := bufio.NewScanner(file)
for scanner.Scan() { fmt.Println(scanner.Text()) }
// Missing: if err := scanner.Err(); err != nil { log.Fatal(err) }

5. Not Flushing Buffered Writer

writer := bufio.NewWriter(file)
writer.WriteString("data")
// writer.Flush()  // Missing! Data may not be written

Practice Questions

1. What is the difference between os.ReadFile and os.Open? ReadFile reads entire file into memory. Open returns a file handle for streaming.

2. When should I use bufio.Scanner? For reading files line-by-line. It handles large files efficiently without loading the entire file.

3. What is the purpose of bufio.Writer.Flush? Flush writes buffered data to the underlying writer. Buffered writes are stored in memory until Flush is called.

4. How do you check if a file exists? os.Stat("file.txt") returns error for non-existent files. Use os.IsNotExist(err) to check.

Challenge: Write a program that reads a CSV file and writes only the first column to a new file.

func main() {
    src, _ := os.Open("input.csv")
    defer src.Close()

    dst, _ := os.Create("output.txt")
    defer dst.Close()

    scanner := bufio.NewScanner(src)
    writer := bufio.NewWriter(dst)
    defer writer.Flush()

    for scanner.Scan() {
        line := scanner.Text()
        columns := strings.Split(line, ",")
        if len(columns) > 0 {
            fmt.Fprintln(writer, columns[0])
        }
    }
}

FAQ

{{< faq question="What is the difference between io.Copy and bufio?" >}} io.Copy streams raw bytes. bufio provides buffering and line-oriented operations. Use io.Copy for binary data, bufio for text. {{< /faq >}}

{{< faq question="How do I append to a file?" >}} Open with os.O_APPEND|os.O_WRONLY flags: os.OpenFile("file.txt", os.O_APPEND|os.O_WRONLY, 0644). {{< /faq >}}

{{< faq question="What is the best way to read a large file?" >}} Use bufio.Scanner for line-oriented, or io.Copy with a buffer for binary. Never os.ReadFile for files larger than memory. {{< /faq >}}

{{< faq question="How do I handle different file encodings?" >}} Use the golang.org/x/text/encoding package for charset conversion. Common encodings: UTF-8, Latin-1, Shift-JIS. {{< /faq >}}

{{< faq question="Can I read from stdin?" >}} Yes. os.Stdin is a file handle. Use bufio.NewScanner(os.Stdin) to read user input. {{< /faq >}}

Try It Yourself

package main

import (
    "bufio"
    "fmt"
    "os"
)

func main() {
    file, err := os.Create("hello.txt")
    if err != nil { fmt.Println(err); return }
    defer file.Close()

    writer := bufio.NewWriter(file)
    fmt.Fprintln(writer, "Hello, World!")
    writer.Flush()

    content, _ := os.ReadFile("hello.txt")
    fmt.Print(string(content))
}

Expected output:

Hello, World!

What's Next

Now that you understand file I/O, explore JSON encoding and decoding in Go.