How to Count Lines in a File Using Go (And the bufio.Scanner Trap You Need to Know)

You are processing server logs in Go.

Your parser uses Scanner to count lines. It works perfectly in testing.

In production, one request writes a 200KB single-line JSON blob into the log. Your line counter stops after the previous line. No panic. No obvious failure. If you forgot to check scanner.Err(), you ship a wrong number.

This is the bufio.Scanner 64KB trap.

It affects many golang count lines utilities that use the default scanner on files with long lines. This guide shows how to avoid it, when bytes.Count is enough, when manual byte scanning is faster, and how to count many files with goroutines without exhausting file descriptors.

If you only need a quick answer without compiling Go code, use the browser-based Line Counter tool. If you are building a Go line counter into a service, CLI, or log processor, keep reading.

Quick Method Guide

For most go count lines file code, use the fixed Scanner version. For golang count lines large file jobs where speed matters, benchmark manual byte scanning. For small files where simplicity matters, bytes.Count is hard to beat.

Method 1: bufio.Scanner - Standard Approach with a Critical Caveat

Most tutorials start here:

package main

import (
	"bufio"
	"fmt"
	"log"
	"os"
)

func countLines(filename string) (int, error) {
	f, err := os.Open(filename)
	if err != nil {
		return 0, err
	}
	defer f.Close()

	scanner := bufio.NewScanner(f)
	count := 0
	for scanner.Scan() {
		count++
	}

	if err := scanner.Err(); err != nil {
		return 0, err
	}
	return count, nil
}

func main() {
	count, err := countLines("data.txt")
	if err != nil {
		log.Fatal(err)
	}
	fmt.Printf("Lines: %d\n", count)
}

The important line is the one many snippets omit:

if err := scanner.Err(); err != nil {
	return 0, err
}

Scanner.Scan() returns false both at EOF and after an error. The only way to know which one happened is scanner.Err(). That is the difference between a reliable golang count lines utility and a quiet undercount.

The 64KB trap

Go's bufio.MaxScanTokenSize is 64 * 1024. The docs also note that the actual maximum token size may be smaller because the buffer may need room for data such as a newline.

Default Scanner uses bufio.ScanLines, so a "token" is a line. If one line is longer than the default maximum, the scanner stops and reports bufio.Scanner: token too long.

Reproduction:

package main

import (
	"bufio"
	"fmt"
	"os"
	"strings"
)

func main() {
	f, err := os.CreateTemp("", "scanner-trap-*.txt")
	if err != nil {
		panic(err)
	}
	defer os.Remove(f.Name())

	longLine := strings.Repeat("x", 100*1024)
	fmt.Fprintln(f, "line 1")
	fmt.Fprintln(f, longLine)
	fmt.Fprintln(f, "line 3")
	f.Close()

	file, err := os.Open(f.Name())
	if err != nil {
		panic(err)
	}
	defer file.Close()

	scanner := bufio.NewScanner(file)
	count := 0
	for scanner.Scan() {
		count++
	}

	fmt.Printf("actual lines: 3\n")
	fmt.Printf("scanner count: %d\n", count)
	fmt.Printf("scanner error: %v\n", scanner.Err())
}

Expected output shape:

actual lines: 3
scanner count: 1
scanner error: bufio.Scanner: token too long

This is why every go bufio scanner 64kb example should show scanner.Err(). The scanner is not silently corrupting data; your code is silent if it ignores the error.

The error text people search for is usually bufio scanner token too long. The fix is not to retry the same scanner; set a larger buffer before scanning.

Fix: configure Scanner.Buffer

Call scanner.Buffer before the first scan.

package main

import (
	"bufio"
	"fmt"
	"log"
	"os"
)

func countLinesRobust(filename string) (int, error) {
	f, err := os.Open(filename)
	if err != nil {
		return 0, fmt.Errorf("open file: %w", err)
	}
	defer f.Close()

	scanner := bufio.NewScanner(f)

	const maxLineSize = 10 * 1024 * 1024
	buf := make([]byte, 64*1024)
	scanner.Buffer(buf, maxLineSize)

	count := 0
	for scanner.Scan() {
		count++
	}

	if err := scanner.Err(); err != nil {
		return 0, fmt.Errorf("scan file: %w", err)
	}
	return count, nil
}

func main() {
	count, err := countLinesRobust("data.txt")
	if err != nil {
		log.Fatal(err)
	}
	fmt.Printf("Lines: %d\n", count)
}

scanner.Buffer(buf, max) sets the initial buffer and the maximum buffer that may be allocated. If a line exceeds max, you will still get bufio.Scanner: token too long. Pick a maximum that matches your input contract.

Use this when:

• You want idiomatic go count lines file code.
• You need to process an io.Reader.
• You may have lines longer than 64KB but still have a reasonable maximum line size.

Production rule for bufio.Scanner:

• bufio.Scanner is excellent for bounded tokens.
• bufio.Scanner is not a raw byte counter.
• bufio.Scanner needs scanner.Buffer when lines may be long.
• bufio.Scanner needs scanner.Err() after every scan loop.
• bufio.Scanner should not be your fastest-path choice for multi-GB raw newline counting.

Avoid this when:

• Lines can be arbitrarily long.
• You only need raw newline counts at maximum throughput.
• You cannot define a safe maximum token size.

For the last case, use manual byte scanning.

Method 2: bytes.Count - The One-Liner Approach

For small known files, read the file and count newline bytes.

package main

import (
	"bytes"
	"fmt"
	"log"
	"os"
)

func countLinesBytes(filename string) (int, error) {
	data, err := os.ReadFile(filename)
	if err != nil {
		return 0, fmt.Errorf("read file: %w", err)
	}
	if len(data) == 0 {
		return 0, nil
	}

	count := bytes.Count(data, []byte{'\n'})
	if data[len(data)-1] != '\n' {
		count++
	}
	return count, nil
}

func main() {
	count, err := countLinesBytes("data.txt")
	if err != nil {
		log.Fatal(err)
	}
	fmt.Printf("Lines: %d\n", count)
}

This is simple and fast, but it reads the entire file into memory. The Go docs describe os.ReadFile exactly that way: it reads the named file and returns the contents.

bytes.Count returns 0 for empty input, and the implementation above returns 0 lines for an empty file. A file containing only \n returns 1.

Use this for scripts and CLIs where files are known to be small. Do not use it as the default golang count lines large file implementation.

Method 3: Manual Byte Scanning - Maximum Performance

Manual byte scanning avoids the Scanner token limit and avoids loading the whole file.

package main

import (
	"fmt"
	"io"
	"log"
	"os"
)

func countLinesFast(filename string) (int, error) {
	f, err := os.Open(filename)
	if err != nil {
		return 0, fmt.Errorf("open file: %w", err)
	}
	defer f.Close()

	buf := make([]byte, 32*1024)
	count := 0
	sawAnyByte := false
	lastByte := byte('\n')

	for {
		n, err := f.Read(buf)
		if n > 0 {
			sawAnyByte = true
			for _, b := range buf[:n] {
				if b == '\n' {
					count++
				}
			}
			lastByte = buf[n-1]
		}
		if err == io.EOF {
			break
		}
		if err != nil {
			return 0, fmt.Errorf("read file: %w", err)
		}
	}

	if sawAnyByte && lastByte != '\n' {
		count++
	}
	return count, nil
}

func main() {
	count, err := countLinesFast("data.txt")
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(count)
}

This is the fastest pattern for many golang count lines large file workloads because it counts bytes directly:

Scanner           -> token creation and scan loop
bytes.Count        -> whole file in memory
manual byte scan   -> fixed buffer, newline byte count

It also handles one line without a trailing newline because it tracks whether any bytes were read and whether the last byte was \n.

Benchmark: Which Method Is Fastest?

Benchmark shape:

func BenchmarkScanner(b *testing.B) {
	for i := 0; i < b.N; i++ {
		_, _ = countLinesRobust("testdata/large.txt")
	}
}

func BenchmarkBytesCount(b *testing.B) {
	for i := 0; i < b.N; i++ {
		_, _ = countLinesBytes("testdata/large.txt")
	}
}

func BenchmarkManual(b *testing.B) {
	for i := 0; i < b.N; i++ {
		_, _ = countLinesFast("testdata/large.txt")
	}
}

Directional results for a 500MB file with about 5 million lines on SSD storage:

The numbers are benchmark-style guidance, not a hardware guarantee. The stable rule is what matters:

• Small known files: bytes.Count.
• Unknown or streaming input: fixed Scanner.
• Huge files or performance-sensitive CLIs: manual byte scanning.

Method 5: Concurrent Line Counting with Goroutines

DevOps and cloud-native Go tools often need to count every log file in a directory. This is where golang count lines goroutine patterns help.

Basic concurrent version:

package main

import (
	"fmt"
	"log"
	"path/filepath"
	"sync"
)

type fileResult struct {
	filename string
	lines    int
	err      error
}

func countLinesDir(dir, pattern string) (map[string]int, int, error) {
	files, err := filepath.Glob(filepath.Join(dir, pattern))
	if err != nil {
		return nil, 0, err
	}

	results := make(chan fileResult, len(files))
	var wg sync.WaitGroup

	for _, file := range files {
		wg.Add(1)
		go func(filename string) {
			defer wg.Done()
			count, err := countLinesRobust(filename)
			results <- fileResult{filename: filename, lines: count, err: err}
		}(file)
	}

	go func() {
		wg.Wait()
		close(results)
	}()

	fileCounts := make(map[string]int)
	total := 0
	for result := range results {
		if result.err != nil {
			log.Printf("warning: %s: %v", result.filename, result.err)
			continue
		}
		fileCounts[result.filename] = result.lines
		total += result.lines
	}

	return fileCounts, total, nil
}

func main() {
	counts, total, err := countLinesDir("/var/log", "*.log")
	if err != nil {
		log.Fatal(err)
	}
	for file, count := range counts {
		fmt.Printf("%s: %d lines\n", file, count)
	}
	fmt.Printf("total: %d lines across %d files\n", total, len(counts))
}

For directories with thousands of files, limit concurrency. Otherwise the program can open too many files at once.

func countLinesDirLimited(dir, pattern string, maxConcurrent int) (int, error) {
	files, err := filepath.Glob(filepath.Join(dir, pattern))
	if err != nil {
		return 0, err
	}

	sem := make(chan struct{}, maxConcurrent)
	results := make(chan int, len(files))
	var wg sync.WaitGroup

	for _, file := range files {
		wg.Add(1)
		go func(filename string) {
			defer wg.Done()

			sem <- struct{}{}
			defer func() { <-sem }()

			count, err := countLinesRobust(filename)
			if err != nil {
				log.Printf("warning: %s: %v", filename, err)
				results <- 0
				return
			}
			results <- count
		}(file)
	}

	go func() {
		wg.Wait()
		close(results)
	}()

	total := 0
	for count := range results {
		total += count
	}
	return total, nil
}

Use a limit such as 8, 16, or 32 depending on disk type, network storage, and file descriptor limits. That is the production-safe golang count lines goroutine pattern.

Part 6: A Production-Ready Line Counter Package

This package counts from any io.Reader, so it works for files, stdin, HTTP bodies, gzip readers, and tests.

package linecounter

import (
	"bufio"
	"bytes"
	"fmt"
	"io"
	"os"
)

type Options struct {
	MaxLineSize int
	CountEmpty bool
	TrimSpace  bool
}

func DefaultOptions() Options {
	return Options{
		MaxLineSize: 10 * 1024 * 1024,
		CountEmpty: true,
		TrimSpace:  false,
	}
}

func CountFile(filename string, opts Options) (int, error) {
	f, err := os.Open(filename)
	if err != nil {
		return 0, fmt.Errorf("linecounter: open %q: %w", filename, err)
	}
	defer f.Close()
	return CountReader(f, opts)
}

func CountReader(r io.Reader, opts Options) (int, error) {
	scanner := bufio.NewScanner(r)

	maxSize := opts.MaxLineSize
	if maxSize <= 0 {
		maxSize = 10 * 1024 * 1024
	}
	buf := make([]byte, 64*1024)
	scanner.Buffer(buf, maxSize)

	count := 0
	for scanner.Scan() {
		line := scanner.Bytes()

		if !opts.CountEmpty && len(line) == 0 {
			continue
		}
		if opts.TrimSpace && len(bytes.TrimSpace(line)) == 0 {
			continue
		}
		count++
	}

	if err := scanner.Err(); err != nil {
		return 0, fmt.Errorf("linecounter: scan: %w", err)
	}
	return count, nil
}

Usage:

count, err := linecounter.CountFile("data.txt", linecounter.DefaultOptions())

opts := linecounter.DefaultOptions()
opts.CountEmpty = false
count, err = linecounter.CountFile("data.txt", opts)

count, err = linecounter.CountReader(req.Body, linecounter.DefaultOptions())

Production checklist:

• Call scanner.Buffer when using Scanner.
• Check scanner.Err() after the loop.
• Close files with defer f.Close().
• Wrap errors with %w.
• Limit goroutine concurrency for many files.
• Avoid os.ReadFile for unknown-size files.

Part 7: Edge Cases and Special Scenarios

stdin

Use the same fixed scanner for stdin. This is the Go equivalent of a wc -l command pipeline.

package main

import (
	"bufio"
	"fmt"
	"os"
)

func main() {
	scanner := bufio.NewScanner(os.Stdin)
	buf := make([]byte, 64*1024)
	scanner.Buffer(buf, 10*1024*1024)

	count := 0
	for scanner.Scan() {
		count++
	}
	if err := scanner.Err(); err != nil {
		fmt.Fprintf(os.Stderr, "error: %v\n", err)
		os.Exit(1)
	}
	fmt.Println(count)
}

For shell workflows, compare this with the wc -l command.

gzip files

The go count lines gzip pattern is: open the file, wrap it in gzip.NewReader, then scan the decompressed stream.

package main

import (
	"bufio"
	"compress/gzip"
	"fmt"
	"os"
)

func countLinesGzip(filename string) (int, error) {
	f, err := os.Open(filename)
	if err != nil {
		return 0, err
	}
	defer f.Close()

	gz, err := gzip.NewReader(f)
	if err != nil {
		return 0, fmt.Errorf("gzip: %w", err)
	}
	defer gz.Close()

	scanner := bufio.NewScanner(gz)
	buf := make([]byte, 64*1024)
	scanner.Buffer(buf, 10*1024*1024)

	count := 0
	for scanner.Scan() {
		count++
	}
	return count, scanner.Err()
}

This go count lines gzip approach does not require writing the decompressed file to disk.

Windows line endings

The default split function is bufio.ScanLines. Go's documentation says it strips a line ending made of one optional carriage return followed by one mandatory newline, written as \r?\n.

That means CRLF files from Windows are handled by default:

scanner := bufio.NewScanner(file)
scanner.Split(bufio.ScanLines) // default; handles CRLF

It does not mean lone \r old-Mac line endings are handled as separate line endings. If you have that input, normalize it or write a custom split function.

Go version notes

• bufio.Scanner.Buffer was added in Go 1.6.
• os.ReadFile was added in Go 1.16; older code used ioutil.ReadFile.
• The examples use %w error wrapping, available since Go 1.13.

Which Go Method Should You Use?

Need to count lines in Go?
|
+-- File is small and known?
|   +-- Use bytes.Count with os.ReadFile
|
+-- Input may be large or streaming?
|   +-- Use Scanner with scanner.Buffer
|
+-- Lines may be longer than your safe Scanner max?
|   +-- Use manual byte scanning
|
+-- Counting many files?
|   +-- Use goroutines with a concurrency limit
|
+-- Counting gzip input?
    +-- Wrap gzip.NewReader and reuse the Scanner helper

The safest default is a fixed Scanner: it is readable, works with io.Reader, and avoids the go bufio scanner 64kb trap when you configure the buffer and check errors.

FAQ

How do I count lines in a file in Go?

Use Scanner, call scanner.Buffer, loop over scanner.Scan(), and check scanner.Err() after the loop. That is the best default go count lines file pattern.

What is the fastest way to count lines in Go?

Manual byte scanning is usually fastest because it counts \n bytes in a fixed buffer without token allocation.

Why does bufio.Scanner give the wrong line count?

Usually because the code ignored scanner.Err(). If a line exceeds the default token limit, Scan() stops and Err() reports bufio.Scanner: token too long, often searched as bufio scanner token too long.

How do I fix bufio.Scanner: token too long?

Call scanner.Buffer(buf, maxSize) before scanning. This is the fix for bufio scanner token too long errors when your maximum line size is known.

How do I count lines in a large file in Go?

Use fixed Scanner or manual byte scanning. Avoid os.ReadFile and bytes.Count for unknown-size or multi-GB files.

How do I count lines in multiple files concurrently in Go?

Use goroutines and a results channel, but add a semaphore or worker pool. That prevents file descriptor exhaustion.

How do I count lines in a gzip file in Go?

Use gzip.NewReader, then pass the decompressed reader to a fixed Scanner. The go count lines gzip code above shows the complete pattern.

Does Go's bufio.Scanner handle Windows line endings?

Yes for CRLF. bufio.ScanLines strips \r?\n, so Windows text files work without special code.

Sources Checked

• Go bufio package documentation for MaxScanTokenSize, Scanner.Buffer, Scanner.Err, Scanner.Scan, and ScanLines: https://pkg.go.dev/bufio
• Go os.ReadFile documentation: https://pkg.go.dev/os#ReadFile
• Go 1.16 release notes for the ioutil.ReadFile to os.ReadFile migration: https://go.dev/doc/go1.16
• Go source for ScanLines CRLF handling: https://go.dev/src/bufio/scan.go

Related Guides and Tools

• wc -l command for shell and stdin workflows.
• Python line counting for scripting, CSVs, and data files.
• Rust line counting for BufReader, read_line, Rayon, and zero-allocation byte scans.
• Java line counting for JVM services and large-file benchmarks.
• Kotlin line counting for useLines, BufferedReader, coroutines, and Android file imports.
• Scala line counting for Source, Using, Files.lines, Spark, and file-handle leak avoidance.
• cross-platform line counting for Linux, macOS, and Windows commands.
• Line Counter tool for no-code line counting.

Need to Count Lines Without Writing Go Code?

If you are debugging a file, validating an export, or just need a quick line count before processing, paste it into the Line Counter. No compilation, no buffer settings, no 64KB surprises.