Data driven transformations: list comprehensions

Originally popularised by the functional language Haskell, list comprehensions give you a different way to code in Python that allows you to focus on the data you're transforming, rather than the functions you use.

Anything you can do with list comprehensions you can achieve using the built-in functions map and filter, but list comprehensions provide you with a simpler to use and more readable syntax. In this article I'll show you the power of list comprehensions by going through some simple examples.

When you're specifying a list in Python, you enumerate your items and surround them with square brackets like so:

>>> wordlist = ['HELLO', 'World', 'how', 'aRe', 'YOU?']

When you use list comprehensions you do the same thing, except that instead of enumerating the contents, you describe the contents as a transformation of another list. Let's take an example: we wanted to take the list of words and make them lowercase. Normally we would use the following:

>>> l = []
>>> for word in wordlist:
        l.append(word.lower())
>>> l
['hello', 'world.', 'how', 'are', 'you?']

But that's too long, and it's slow, since Python has to implicitly loop whenever we use a for statement. If we were used to functional programming we could write instead:

>>> import string
>>> map(string.lower,wordlist)
['hello', 'world.', 'how', 'are', 'you?']

That's an improvement, but it's a little cryptic. By using a list comprehension instead we could write:

>>> [word.lower() for word in wordlist]
['hello', 'world.', 'how', 'are', 'you?']

The second version isn't any shorter, but it can be clearer in many circumstances. You'll have to decide for yourself which approach suits your circumstances.

List comprehensions can also be used to completely replace the built-in filter function, for example if you wanted only the words in the list that were already lowercase you could write:

>>> [word for word in wordlist if world.islower()]
['how']

map and filter are the cornerstones of functional programming, using list comprehensions you can use either or both in an intuitive way. If we wanted to combine both map and filter in one expression it is as simple as:

>>> [word.lower() for word in wordlist if not world.islower()]
['hello', 'world.', 'are', 'you?']

You can nest list comprehensions either in the first section like so:

>>> vowels = ['a','A','e','E','i','I','o','O','u','U']]
>>> [[letter for letter in word if letter not in vowels] for word in wordlist]
[['H', 'L', 'L'], ['W', 'r', 'l', 'd'], ['h', 'w'], ['R'], ['y', '?']]

Or in the last section, allowing you to chain multiple tranformations together:

>>> [a.lower() for a in [b[i] for b in x for i in range(len(b))]]
['h', 'e', 'l', 'l', 'o', 'w', 'o', 'r', 'l', 'd', 'h', 'o', 'w', 'a', 'r', 'e', 'y', 'o', 'u', '?']


As you can see, when you start chaining list comprehensions together things get complicated. When you have more than one list comprehension in an expression it's a good idea to break things up on to multiple lines. You don't always need to nest list comprehensions though, if all you're doing is iterating over multiple lists you can include them in the same list comprehension.

In the following example we want to generate all permutations of guests to a dinner party, we can do it with one simple list comprehension:

>>> guests = ['Chris', 'Brendan', 'Jimmy', 'Mel', 'Mike', 'Jess']
>>> [(seat1, seat2) for seat1 in guests for seat2 in guests if seat1 != seat2]
[('Chris', 'Brendan'), ('Chris', 'Jimmy'), ('Chris', 'Mel'), ('Chris', 'Mike'), ('Chris', 'Jess'), ('Brendan', 'Chris'), ('Brendan', 'Jimmy'), ('Brendan', 'Mel'), ('Brendan', 'Mike'), ('Brendan', 'Jess'), ('Jimmy', 'Chris'), ('Jimmy', 'Brendan'), ('Jimmy', 'Mel'), ('Jimmy', 'Mike'), ('Jimmy', 'Jess'), ('Mel', 'Chris'), ('Mel', 'Brendan'), ('Mel', 'Jimmy'), ('Mel', 'Mike'), ('Mel', 'Jess'), ('Mike', 'Chris'), ('Mike', 'Brendan'), ('Mike', 'Jimmy'), ('Mike', 'Mel'), ('Mike', 'Jess'), ('Jess', 'Chris'), ('Jess', 'Brendan'), ('Jess', 'Jimmy'), ('Jess', 'Mel'), ('Jess', 'Mike')]


It's not going to replace everything in your programs, but add list comprehensions to your Python toolbox and you'll soon see that they can make your programs smaller, clearer and faster than they were previously, and since they concentrate on data rather than process, they're easy to write -- even when what you need to do is quite complicated.