IEnumerable is awesome

I’ve always thought that one of the most underrated features of C# is the yield statement and its companion, IEnumerable<T>. It may be because I’m working a lot with people coming from an exclusively C++ background – it takes some time to adapt to a new language with new paradigms, especially when that language can look a lot like “C++ without delete!” at first. But there are so many wonderful constructs in C# (especially in the past couple versions) that it’s a shame when people keep writing code “the old way”…

That’s why I’m going to write a few “101” articles I can refer my co-workers to (hi co-worker!).

It starts (after the jump) with how yield+IEnumerable is awesome.

The yield statement

The yield statement allows you to write a pseudo¹ coroutine specifically for generating enumerable objects (which is why it is sometimes called a “generator”). The first beauty of it is that you don’t have to create the enumerable object itself – it is created for you by the compiler. The second beauty is that this generator will return each item one by one without storing the whole collection, effectively “lazily” generating the sequence of items.

To illustrate this, look at how the following piece of code will generate the all time favorite Fibonacci sequence.

public static IEnumerable<int> GetFibonacciSequence()
{
    yield return 0;
    yield return 1;

    int previous = 0;
    int current = 1;

    while (true)
    {
        int next = previous + current;
        previous = current;
        current = next;
        yield return next;
    }
}

The awesome thing is that it’s an infinite loop! It would obviously never return if the function didn’t behave like a coroutine: it returns the next number to the caller and only resumes execution when the caller asks for another number. It’s like you’re “streaming” the Fibonacci sequence!

You can stop any time you want. For instance:

int count = 20;
foreach (int i in GetFibonacciSequence())
{
    Console.WriteLine(i);
    if (--count == 0)
        return;
}

Or, even better, using some of LINQ’s extension methods:

using System.Linq;
foreach (int i in GetFibonacciSequence().Take(20))
{
    Console.WriteLine(i);
}

Performance Gains

There are many advantages to using the yield statement, but most C++ programmers are not really swayed by arguments of coding terseness and expressivity, especially when it involves “black magic” going on inside the compiler²: they usually mostly, err, yield to performance related arguments (see what I did, here?).

So let’s write a simple program that generates many “widgets”, each 256 bytes in size, and print the peak memory usage:

class Widget
{
    private byte[] mBuffer;

    public Widget(int size)
    {
        mBuffer = new byte[size];
    }
}

class Producer
{
    // The old, classic way: return a big array. Booooorriiiiing.
    public IEnumerable<Widget> ProduceArray(int count, int widgetSize)
    {
        var widgets = new Widget[count];
        for (int i = 0; i < count; i++)
        {
            widgets[i] = new Widget(widgetSize);
        }
        return widgets;
    }
    
    // The new funky, trendy, hipstery way! Yieldy yay!
    public IEnumerable<Widget> ProduceEnumerable(int count, int widgetSize)
    {
        for (int i = 0; i < count; i++)
        {
            yield return new Widget(widgetSize);
        }
    }
}

class Program
{
    static void Main(string[] args)
    {
        int size = 256;
        int count = 1000000;
        ProduceAndPrint(false, count, size);    // LINE 7
        Console.WriteLine("Generated {0} widgets of size {1} (total size = {2})", count, size, count * size);
        Console.WriteLine("Memory Peaks:");
        var process = Process.GetCurrentProcess();
        Console.WriteLine("Virtual MemoryttPaged MemoryttWorking Set");
        Console.WriteLine("{0} Mbtt{1} Mbtt{2} Mb", process.PeakVirtualMemorySize64 / 1024, process.PeakPagedMemorySize64 / 1024, process.PeakWorkingSet64 / 1024);
    }

    static void ProduceAndPrint(bool useEnumerable, int count, int widgetSize)
    {
        var producer = new Producer();
        if (useEnumerable)
        {
            int i = 0;
            foreach (var w in producer.ProduceEnumerable(count, widgetSize))
            {
                ++i;
            }
        }
        else
        {
            int i = 0;
            foreach (var w in producer.ProduceArray(count, widgetSize))
            {
                ++i;
            }
        }
    }
}

This prints, on average:

Generated 1000000 widgets of size 256 (total size = 256000000)
Memory Peaks:
Virtual Memory          Paged Memory            Working Set
488572 Mb               299544 Mb               293100 Mb

Now, on line 7, change false into true. This will make the Producer return a yield-enumerated bunch of widgets, instead of returning a big array. This prints:

Generated 1000000 widgets of size 256 (total size = 256000000)
Memory Peaks:
Virtual Memory          Paged Memory            Working Set
133564 Mb               14156 Mb                12984 Mb

Woohoo! That’s almost 4 times less virtual memory used, and a 25 times smaller working set! All of this because the garbage collector can now kick in during the enumeration and discard some of the widgets we’re not using anymore. You can actually print the number of garbage collections triggered using GC.CollectionCount(0) (all those widgets will be in generation 0 because they’re short lived). In the first case (returning the full array of widgets) I usually get 49 collections. For the second case (returning widgets one by one) I usually get 66 collections.

Of course, you may get frightened by all those garbage collections that would slow down your loop (C++ programmers are easily scared by garbage collectors), and that’s a legitimate concern if you’re writing a real-time ultra-sensitive application (although you’d have to check first that it would indeed be a problem). But for other kinds of applications, it can be a real life-saver – like for instance when you’re fetching huge amounts of data from a database of some sorts, and although each piece of data fits in memory, the whole batch doesn’t.³

More!

If you want more meaty stuff on all this, I recommend reading Raymond Chen’s series: part 1, part 2, part 3 and part4. You can also have a look at the fun traps that await you as told by Eric Lippert: part 1 and part 2.