Iterators & Generators

Iterators are containers for objects which allow to loop over them using a “for” loop. In contrast to lists, they don’t store the objects in a specific order (you can’t pull one out from the middle), but yields those objects in a specific order, often creating them on the fly, when needed. That is the purpose of a generator function.

As an effect, they are much smaller in size than comparable lists (one million integer, for instance, don’t need to be in memory at once, but one by one).

list: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
iterator: [generator function: count from 1 to 10, step-size 1]

  • iterators are much faster to create than comparable lists

  • iterators can be of unknown length, whereas the length of a list is always given

Iterable:

Any object which can be looped over using a for loop e.g.

for item in some_iterable:
    print(item)

Any iterable object must implement the __iter__() method, which must return an Iterator object, which defines, how to loop over these particular object type in a for-loop.

class MyIterable:

    def __init__(self, values: list):
        self.values = values

    def __iter__(self):
        return MyIterator(self)
Iterator:

Objects that define how to loop over a sequence of Iterables.

For instance, not all Iterable objects support indexing (0, 1, 2,…) like a list, but follow a different logic when they’re used in a for-loop.

An Iterator must implement these methods:

  • __init__(self) … called upon creation e.g. MyIterator(). It commonly accepts the items, it is supposed to iterate over (and initializes an index, if items support indexes):

    def __init__(self, items):
    self._items = items
    self._index = 0

  • __iter__(self) … called when running iter(object). It must return an Iterator, so itself:

    def __iter__(self):
    return self

  • __next__(self) … called when running next(object). It returns an Iterable object and defines a logic to decide, which object is returned. If no object are left to produce, it commonly returns a StopIteration exception.

    def __next__(self):
    if(self._index < len(self._items)):
        result = self._item(self._index)
        index += 1
        return result
    raise StopIteration

Important

An Iterator is also an Iterable, which means you can loop over iterators:

numbers = [1, 2, 3]
iterator1 = iter(numbers)
iterator2 = iter(iterator1)
print(iterator1 is iterator2) # True

Definition

  1. Anything that can be passed to iter without a TypeError is an iterable

  2. Anything that can be passed to next without a TypeError is an iterator

  3. Anything that returns itself when passed to iter is an iterator

Generator:

A generator is a function which returns an Iterator of passed in Iterables. The generator function uses the yield statement, which puts the function on pause between calls from the next() function.

def square_all(numbers):
    for n in numbers:
        yield n**2

Generators are considered an easier, more common way to create Iterators objects, as defining an entire Iterator class (see above). The generator function commonly implements some logic which are applied to the Iterable items before yielding them. In case the Iterable does not support indexing, an index must be set to the next item manually.

The same iterator can also be implemented using a generator expression:

def square_all(numbers):
    return (n**2 for n in numbers)

List comprehension statements can be transformed into a generator expression by replacing the square brackets:

lines = [line.strip("\n") for line in poem_file if line != "\n"] # list comprehension
lines = (line.strip("\n") for line in poem_file if line != "\n") # generator expression

Both generators functions and generator expressions return an iterator object. Generator functions can be replaced by a generator expression:

def get_a_generator(some_iterable):
    for item in some_iterable:
        if some_condition(item):
            yield item

def get_a_generator(some_iterable):
    return(item for item in some_iterable if some_condition(item))

A generator expression always has common brackets (“()”) around it. When using square brackets (“[]”) then the return type is not a generator, but a simple list (with all possible values inside):

return (n**2 for n in favourite_numbers) # returns generator
return [n**2 for n in favourite_nunbers] # returns list

yield_from in generator functions:

for num in range(start, stop):
    yield num

# same as
yield from range(start, stop)

Example

This example shows the use of an custom Iterator TeamIterator class looping over an Iterable Team class, which contains of Member objects, which are looped over:

class Member:

    def __init__(self, name):
        self.name = name


class Team:

    def __init__(self):
        self.members = list()

    def add_member(self, member):
        self.members.append(member)

    def __iter__(self):
        return TeamIterator(self)


class TeamIterator:

    def __init__(self, team):
        self._team = team
        self._index = 0

    def __iter__(self):
        return self

    def __next__(self):
        if(self._index < len(self._team.members)):
            result = self._team.members[self._index]
            self._index += 1
            return result
        raise StopIteration


team = Team()
team.add_member(Member("Frank"))
team.add_member(Member("Lucy"))
team.add_member(Member("Ole"))

team_iterator = iter(team)
while True:
    try:
        member = next(team_iterator)
        print(member.name)
    except StopIteration:
        break

which outputs:

Frank
Lucy
Ole

Sources

https://thispointer.com/python-how-to-make-a-class-iterable-create-iterator-class-for-it/ https://treyhunner.com/2019/06/loop-better-a-deeper-look-at-iteration-in-python/#Generators_are_iterators