Modify iterables while iterating in Python
Redowan Delowar
March 4, 2022
If you try to mutate a sequence while traversing through it, Python usually doesn't
complain. For example:
The above snippet iterates through a list of numbers and modifies the list l in-place to
remove any even number. However, running the script prints out this:
Wait a minute! The output doesn't look correct. The final list still contains 56 which is
an even number. Why did it get skipped? Printing the members of the list while the for-loop
advances reveal what's happening inside:
From the output, it seems like the for-loop doesn't even visit all the elements of the
sequence. However, trying to emulate what happens inside the for-loop with iter and next
makes the situation clearer. Notice the following example. I'm using ipython shell to
explore:
The REPL experiment reveals that:
> Whenever you remove an element of an iterable that's already been visited by the iterator,
> in the next iteration, the iterator will jump right by 1 element. This can make the
> iterator skip a value. The opposite is also true if you prepend some value to a sequence
> after the iterator has started iterating. In that case, in the next iteration, the
> iterator will jump left by 1 element and may visit the same value again.
Here's what happens when you prepend values after the iteration has started:
Notice how the element 4 is being visited twice after prepending a value to the list l.
Solution
To solve this, you'll have to make sure the target elements don't get removed after the
iterator has already visited them. You can iterate in the reverse order and remove elements
maintaining the original order. The first snippet can be rewritten as follows:
Running the script prints:
Notice, how the iterator now visits all the elements and the final list contains the odd
elements as expected.
Another way you can solve this is - by copying the list l before iterating. But this can
be expensive if l is large:
This time, the order of the iteration and element removal is the same, but that isn't a
problem since these two operations occur on two different lists. Running the snippet
produces the following output:
What about dictionaries
Dictionaries don't even allow you to change their sizes while iterating. The following
snippet raises a RuntimeError:
You can solve this by making a copy of the keys of the dictionary and iterating through it
while removing the elements from the dictionary. Here's one way to do it:
Running the snippet prints:
Voila, the key-value pairs of the even numbers have been removed successfully!
Further reading
- [How to modify a list while iterating - Anthony Sottile]
[how to modify a list while iterating - anthony sottile]:
https://www.youtube.com/watch?v=JXis-BKRDFY
Discussion in the ATmosphere