Python is known for allowing you to produce elegant, simple code and reads almost as well as plain English. List comprehension is one of the language’s most distinctive features, allowing you to develop sophisticated functionality with just one line of code. On the other hand, many Python developers struggle to utilize the more advanced aspects of list comprehension fully. Some programmers may overuse them, resulting in less efficient and difficult-to-read code.
By the end of this article, we are sure that you’ll have a solid understanding of Python list comprehensions and how to make the most of their features. You’ll also learn about the trade-offs that come with using them so that you can figure out when other ways are better.
The following are our key concerns in this article:
- In Python, rewrite loops and map() calls list comprehension.
- You can use comprehensions, loops, or map() calls.
- Conditional logic will boost your understanding.
- Replace the filter() with comprehensions
- To answer performance questions, profile your code.
Lists in Python: How to Make Them
In Python, you can make lists in several different ways. Let’s look at making lists in these ways to understand better the trade-offs of utilizing list comprehension in Python.
Using for Loops
The for loop is the most popular sort of loop. A for loop is used to make a list of elements in three steps:
- Create a new empty list.
- Loop through an iterable or a collection of components.
- Each element is added to the end of the list.
These stages are completed in three lines of code if you wish to construct a list of the first ten perfect squares:
square_vals = [] for i in range(10): square_vals.append(i * i) print(square_vals)
You create an empty list, squares, here. Then, you use a for loop and pass the value as a parameter to iterate across the range.
Finally, you add the result to the end of the list by multiplying each integer by itself.
Utilizing map() Objects
The map() method offers a functional programming-based alternative. When you give map() a function and an iterable, it creates an object. This object stores the results of executing each iterable element via the provided process. Consider the following scenario in which you must calculate the price after tax for a series of transactions:
transactions = [11.09, 33.56, 67.84, 14.56, 16.78] TAX_RATE = .058 def get_price_with_tax(transaction): return transaction * (1 + TAX_RATE) total_prices = map(get_price_with_tax, transactions) list(total_prices)
There’s an iterable ‘transactions’ here, as well as a method get_price_with_tax(). You call map() with these inputs and save the result in total_prices. Further, you can easily convert this map object into a list using a list().
Making Use of List Comprehensions
A third technique to make lists is to use list comprehensions. You could also endeavor to rewrite the for loop from the first example in just one line of code using this elegant approach:
square_vals = [i * i for i in range(10)] print(square_vals)
Rather than starting with an empty list and adding each element at the end, use this approach to define the list and its contents at the same time:
new_list = [expression for member in iterable]
In Python, every list comprehension has three components:
- A valid expression that returns a value is the member itself, a call to a method, or another acceptable expression.
The square of the member value is represented by the expression i * i in the example above. - The item or value in the list or iterable is called a member. The member value in the case above is i.
- A list, sequence, generator, set, or any other object that may return its elements one by one is known as an iterable. The iterable in the preceding example is of range 10.
Because the expression requirement is flexible, list comprehension in Python is used in various situations where you would rather use a map(). The price example can be rewritten with its list comprehension as follows:
transactions = [1.09, 23.56, 57.84, 4.56, 6.78] TAX_RATE = .08 def get_price_with_tax(transaction): return transaction * (1 + TAX_RATE) total_prices = [get_price_with_tax(i) for i in transactions] print(total_prices)
The only difference between this implementation and map() is that list comprehension returns a list rather than a map object in Python.
List Comprehensions Advantages
Loops and maps() are typically described as less Pythonic than list comprehensions. But, rather than accepting that judgment at face value, it’s worthwhile to consider the advantages of utilizing a list comprehension in Python over the alternatives. You’ll learn about a couple of cases where the other options are a better later on.
List comprehensions are used for mapping, filtering, and essential list generation. One of the crucial advantages of utilizing a list comprehension in Python is that it is a single tool used in various situations. In fact, you don’t need to adopt a new strategy for each case.
List comprehensions are regarded as Pythonic, as Python embraces simple, powerful tools used in several scenarios. As a bonus, you won’t have to remember the appropriate order of parameters when using a list comprehension in Python, as you would when calling map().
List comprehensions are easier to read and grasp than loops since they are more declarative. You must manually build an empty list, loop through the entries, and add each to the list’s end. Instead, using a list comprehension in Python, you can concentrate on what you want to put in the list and trust Python to handle the list generation. It would help to focus on how the list is constructed while using loops.
How to increase your Comprehension Power
It’s helpful to grasp the variety of functionality that list comprehensions can provide to appreciate their significance fully. You’ll also want to be aware of the changes coming in Python 3.8 and above to list comprehension.
Conditional Logic
You saw this algorithm for creating list comprehensions earlier:
new_list = [expression for member in iterable]
While this formula is correct, it is also a little lacking. Optional conditionals are now supported in a more detailed description of the comprehension formula. Adding a conditional to the end of an expression is the most popular approach to add conditional logic to a list comprehension:
resultant_new_list = [expression for member in iterable (if conditional)]
Your conditional statement appears exactly before the closing bracket in this example. Conditionals are useful because they allow list comprehensions to filter out undesired values without having to use the filter() function:
sentence = 'codeunderscored is the epitome of coding success' vowel_chars= [i for i in sentence if i in 'aeiou'] print(vowel_chars)
The conditional statement in this code block filters out any characters in the sentence that aren’t vowels. Any valid expression can be tested using the conditional. You can even relocate the conditional logic to a new function if you require a more detailed filter:
sentence = 'The rocket, who was named codeunderscored, came back from Mars because he missed his coding friends.' def check_if_is_consonant(new_char): vowel_chars = 'aeiou' return new_char.isalpha() and new_char.lower() not in vowel_chars consonants = [i for i in sentence if check_if_is_consonant(i)] print(consonants)
You design a sophisticated filter called check_if_is_consonant() and give it to your list comprehension as a conditional statement—the member value i is also supplied to your function as an argument.
For easy filtering, you can put the conditional at the end of the statement, but what if you want to update a member value instead of filtering it out? It’s best to put the conditional near the beginning of the expression in this case:
resultant_new_list = [expression (if conditional) for member in iterable]
You can apply conditional logic with this formula to choose from various output alternatives. If you have a list of prices, for example, you might want to replace negative prices with 0 while leaving the positive numbers alone:
previous_prices = [11.25, -19.45, 10.22, 13.78, 5.92, -11.16] current_prices = [i if i > 0 else 0 for i in previous_prices] print(current_prices)
If i > 0 else 0, your expression i contains a conditional statement. If the number is positive, Python should print the value of i, but if the number is negative, Python should modify i to 0. If all of this is too much for you, consider conditional logic as a separate function:
def get_price(price): return price if price > 0 else 0 current_prices = [get_price(i) for i in previous_prices] print(current_prices)
get_price() now has your conditional statement, which you can use as part of your list comprehension expression.
Using Comprehensions from a Set and a Dictionary
In Python, you can create set and dictionary comprehensions in addition to list comprehensions.
In Python, a set comprehension is nearly identical to a list comprehension. On the other hand, setting list comprehensions ensures that the output is free of duplicates. Instead of brackets, curly braces can be used to create a set comprehension:
code_val = "code, uh, finds a pattern" unique_vowels = {i for i in code_val if i in 'aeiou'} print(unique_vowels)
All of the distinctive vowels discovered in code_val are output by your set comprehension. Sets, unlike lists, do not guarantee that items are preserved in the order you choose. That is why, even though the initial vowel in code_val is ‘i’, the first member of the set is a.
Comprehensions from dictionaries are identical, with the addition of defining a key:
square_vals = {i: i * i for i in range(10)} print(square_vals)
Curly braces ({}) and a key-value pair ( i: i * i) are used in your expression to generate the squares dictionary.
Making Use of the Walrus Operator
The assignment expression, often known as the walrus operator, will be introduced in Python 3.8. Consider the following example to see how you may put it to use.
Let’s say you need to send ten queries to an API to get temperature data. Assume that each request will yield extraordinary results. You only want to see temperatures above 100 degrees Fahrenheit in your results. In this circumstance, there is no method to fix the problem using a list comprehension in Python.
The conditional in the formula expression for the member in iterable (if conditional) has no mechanism of assigning data to a variable that the expression can access.
The walrus operator solves this problem. It enables you to execute an expression while assigning the result to a variable. The following example demonstrates how this is possible when using get_weather_data() to generate phony weather data:
import random def get_weather_data(): return random.randrange(110, 130) list_hot_temperatures = [temp for _ in range(20) if (temp := get_weather_data()) >= 100] print(list_hot_temperatures)
Although you won’t need to utilize the assignment expression inside of list comprehension in Python very often, it’s a handy tool to have.
When to Avoid Using Python’s List Comprehension
List comprehensions help write elegant, easy-to-read, and debug code, but they aren’t the best solution in every situation. They may cause your code to execute slower or consume more memory. It’s probably advisable to choose an alternative if your code is less performant or challenging to understand.
Be Wary of Nested Comprehensions
Comprehensions can be nested within a collection to build combinations of lists, dictionaries, and sets. Assume a climate laboratory monitors the high temperatures in five cities during the first week of June. A list comprehension nested within a dictionary comprehension could be the ideal data structure for storing this information:
computers = ['HP', 'Dell', 'Chrome Book', 'Apple', 'IBM'] comp_list = {comp: [0 for _ in range(7)] for comp in computers} print(comp_list)
A dictionary comprehension is used to create the outer collection temperatures. Another comprehension is contained in the expression, which is a key-value pair. This code will generate a list of data for each city in cities in a short amount of time.
Matrixes are frequently used in mathematics, and nested lists are common to generate them. Examine the code block example below:
matrix_vals = [[i for i in range(2)] for _ in range(3)] print(matrix_vals)
The outer list comprehension [… for _ in range(3)] generates three rows, whereas the inner list comprehension [i for i in range(2)] assigns values to each of these rows. So far, each nested comprehension’s purpose has been rather self-evident.
However, in some cases, such as flattening nested lists, the reasoning may make your code more difficult to understand.
Take this example of flattening a matrix with a stacked list comprehension:
matrix_vals = [ [0, 0, 0], [1, 1, 1], [2, 2, 2], ] flat_vals = [num for row in matrix_vals for num in row] print(flat_vals)
The code for flattening the matrix is short, but it may not be easy to understand how it works. If you use for loops to flatten the same matrix, on the other hand, your code will be much more straightforward:
matrix_vals = [ [0, 0, 0], [1, 1, 1], [2, 2, 2], ] flat_vals = [] for row in matrix_vals: for val in row: flat_vals.append(val) print(flat_vals)
As you can see, the code explores the matrix one row at a time, extracting all of the elements in that row before going on to the next.
While the single-line nested list comprehension may appear more Pythonic, the essential thing is to build code that your team can easily understand and alter. You’ll have to make a decision based on whether you think comprehension helps or damages readability when you choose your technique.
Why use use generators when working with large datasets?
In Python, a list comprehension loads the whole output list into memory. It is usually fine for minor or even medium-sized lists. A list comprehension will admirably solve this problem if you wish to sum the squares of the first one thousand integers.
sum([val * val for val in range(3000)])
What if you needed to add the squares of the first billion integers? If you tried them on your computer, you might have noticed that it gets unresponsive. Python is attempting to generate a list containing one billion numbers, which will take up more memory than your computer can handle. Your computer may lack the resources required to build and store an extensive list in memory. If you go ahead and do it anyway, your computer may slow down or even crash.When the size of a list becomes a concern, using a generator rather than a list comprehension in Python is generally preferable.
A generator returns an iterable rather than creating a single prominent data structure in memory. While only holding a single value at a time, your code can ask for the following item from the iterable as many times as necessary or until you reach the end of your sequence.
If you use a generator to add the first billion squares, your software will probably run for a while, but it won’t cause your machine to freeze. A generator is used in the following example:
sum(i * i for i in range(1000000000))
You can know it’s a generator because brackets or curly braces don’t enclose the expression. Parentheses surround generators if desired.
Although the example above still involves a lot of work, it does so in a lazy manner. Calculation of values only happens when requested due to lazy evaluation. Following a value generation (for example, 267 * 267), the generator can add that value to the ongoing sum, discard that value, and generate the following value (268 * 268).
The cycle begins again when the sum function wants the next value. This method maintains the RAM footprint to a minimum.
Because map() is a lazy function, memory will not be an issue if you use it in this case:
sum(map(lambda i: i*i, range(1000000000)))
Optimize Performance by Profiling
So, which method is the quickest? Is it advisable to use list comprehensions or one of their substitutes? Rather than sticking to a single guideline that applies to all situations, it’s better to ask yourself if performance matters in your particular case. If not, it’s usually advisable to go with the strategy that produces the most precise code!
It’s usually preferable to profile several ways and listen to the data if you’re in a situation where performance is critical. timeit is a handy library for calculating how long portions of code take to execute. To compare the runtimes of map(), for loops, and list comprehensions, use timeit:
Conclusion
Python pushes programmers and developers to write efficiently, easy to understand, and almost as easy to read code. The python list and list compression features are two of the language’s most distinguishing features, which are used to build significant functionality with just one line of code.
List comprehensions are used for constructing new lists from other iterables like tuples, strings, arrays, lists, etc. A list comprehension comprises brackets that hold the expression run for each element and a for loop that iterates through each element.