C# – How to change StreamWriter’s buffer size

by

StreamWriter buffers by writing data to an internal char array with a default size of 1024 (and a minimum size of 128). Once the buffer is full (or when you dispose the StreamWriter), it flushes the buffer to the underlying stream.

You can change StreamWriter’s buffer size by passing in the bufferSize parameter. Here’s an example of increasing the buffer size to 128 KB and writing a big file (~500 MB):

using System.IO;

var filePath = @"D:\BigFiles\abc.txt";

using (var writer = new StreamWriter(filePath, true, Encoding.UTF8, bufferSize: 128 * 1024))
{
    for (int i = 0; i < 40_000_000; i++)
    {
        writer.WriteLine("hello world");
    }
}
Code language: C# (cs)

Using StreamWriter is a lower-level alternative to writing a file when you want more control over the process.

In this example, increasing the buffer size from 1 KB to 128 KB made it write the file 2x faster (on my specific machine). In general, when you’re writing a large amount of data, increasing the buffer size will speed things up, but only up to a certain point. For example, doubling the buffer size from 128 KB to 256 KB didn’t help at all (but it did waste memory). To find the optimal buffer size in your specific situation, you can try different buffer sizes and measure performance.

FileStream’s buffer size

FileStream has its own buffer. When you write to a FileStream (directly, or when StreamWriter writes to it), it buffers the data in a byte array with a default size of 4096 and then flushes the data to disk once the buffer is full.

When you try to write more data than the FileStream buffer can hold, it skips the buffer and writes to disk. This means if you change the StreamWriter to have a larger buffer size than FileStream (default of 4096), there’s no need to also change FileStream’s buffer size.

If you’re directly writing to FileStream (instead of using StreamWriter), you can change its buffer size in the constructor:

using System.IO;

var filePath = @"D:\BigFiles\abc.txt";

using (var stream = new FileStream(filePath, 
    FileMode.Append, FileAccess.Write, FileShare.None,
    bufferSize: 128 * 1024))
{
    //write to stream
}
Code language: C# (cs)

Using Flush() and AutoFlush

You can flush the buffers before waiting for them to get full. For example, you could be writing messages periodically to a file while holding the file open (like logging) and want the data to be written to disk immediately.

There are two ways to do this:

  • Call StreamWriter.Flush() to manually flush the buffers as needed.
  • Set StreamWriter.AutoFlush=true to automatically flush the buffers after every write.

These flush both buffers – StreamWriter’s and the stream’s. In other words, if you’re writing to a file stream, calling Flush() will write the buffered data to disk.

Here’s an example of using Flush() after writing a few messages:

using (var writer = new StreamWriter(filePath))
{
    writer.WriteLine("1 - Starting");
    writer.WriteLine("2 - Getting data");
    writer.Flush();

    //and then write more messages later on 
    writer.WriteLine("3 - Starting");
    writer.WriteLine("4 - Getting data");
    writer.Flush();
}
Code language: C# (cs)

Here’s an example of using AutoFlush:

using (var writer = new StreamWriter(filePath))
{
    writer.AutoFlush = true;

    writer.WriteLine("1 - Starting");
    writer.WriteLine("2 - Getting data");
    writer.WriteLine("3 - Starting");
    writer.WriteLine("4 - Getting data");
}
Code language: C# (cs)

This is useful if you want to flush after every write and don’t want to litter the code with several calls to Flush().

Don’t use AutoFlush when writing a large amount of data

The purpose of buffering is to minimize IO operations when writing a large amount of data. When you use AutoFlush, you essentially remove buffering and maximize the number of IO operations and therefore tanks performance. AutoFlush is useful in some scenarios, but don’t use it when writing a large amount of data.

As a concrete example, using AutoFlush and writing a 500 MB file took 230x longer (a few minutes!) compared to writing the same amount of data with buffering (using the code shown at the beginning of this article).

Related Articles