How to Split List Into Even Chunks in Python

Lists are popular data structures used by almost every Python developer. Sometimes, you may need to split a list into evenly sized chunks or N size chunks in Python. There are several ways to easily do it. In this article, we will learn how to split Python list into N sized chunks.

Split Python List into N Size Chunks

We will look at 5 different ways to split list into N size chunks in Python. Let us say you have the following Python lists.

lst = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
lst1 = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

Let us say you want to split these lists into evenly sized chunks of 2 items each. Let us look at the different ways to do this.

1. Using Yield Keyword

Yield keyword is an important feature in Python programming language. It returns a generator iterator object instead of returning all the actual values. This iterator is like a pointer to the generator function. Every time the iterator is accessed, it returns the next value of the sequence using the generator function, instead of returning the entire sequence at once. Since it does not store or return all the values, except the iterator, it is very memory efficient and fast. Here is how to use yield keyword to split our list.

def split_list(l,n):
    for i in range(0, len(l),n):
        yield l[i:i+n]

lst = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
chunk_size=2
chunks=list(split_list(lst,chunk_size))
print(chunks)

# Output
[[1, 2], [3, 4], [5, 6], [7, 8], [9, 10]]

If your list has some elements that are left after splitting the original list, they are put into a separate list. Let us say your list has 11 items and you want to split them into smaller lists of 2 items each, then here is the output you will get.

lst1 = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
split_list(lst1,n)

# Output
[[1, 2], [3, 4], [5, 6], [7, 8], [9, 10], [11]]

Using yield keyword is really useful when you have large lists which can take up large amount of memory. In all use cases, it is one of the fastest and memory efficient ways to split lists.

2. Using List Comprehension

Understanding what yield keyword does can be difficult. In this case, you can use list comprehensions to split your list. List comprehensions are mainly created to provide a simple, easy way to create a list from another iterable object such as lists, tuples, etc.

Here is a simple example to illustrate it.

chunk_size=2
chunks = [lst[i:i + chunk_size] for i in range(0, len(lst), chunk_size)]
print(chunks)

Here is the output you will get.

[[1, 2], [3, 4], [5, 6], [7, 8], [9, 10]]

Let us look at the expression used for list comprehension in detail. The range function creates a list of indexes, one for each list item, based on original list items. The for loop basically loops through this list of by skipping 2 (chunk_size) items at a time. So the value of i in above expression is 0, 2, 4, …

In each iteration, we slice and return a chunk of original list such that the first item is the loop index i of for loop, and the number of items in the chunk is equal to chunk size. The chunks returned are [1, 2], [3, 4]…

If your original list contains leftover items, they will be put in a separate list.

lst1 = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
chunk_size=2
chunks = [lst[i:i + chunk_size] for i in range(0, len(lst), chunk_size)]
print(chunks)

# Output
[[1, 2], [3, 4], [5, 6], [7, 8], [9, 10], [11]]

You can also add the above code to create a function.

def split_list(l, n):
    chunks = [l[i:i + n] for i in range(0, len(l), n)]
    print(chunks) 

split_list(lst,2)

Using list comprehensions is a very compact and elegant way to split lists. They were specifically created for this purpose.

3. Using itertools

The above 2 methods do not use any library. In this method, we will use itertools library, which provides many useful functions to work with iterables such as lists. It provides islice() function for this purpose. It is already present in every Python installation so you do not need to install it separately.

from itertools import islice
lst = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
def split_list(l, chunk_size):
     l = iter(l)
     return iter(lambda: tuple(islice(l, chunk_size)), ())

list(split_list(lst,2))
[(1, 2), (3, 4), (5, 6), (7, 8), (9, 10), (11,)]

In the above code, we first import islice() function. We then define split_list() function that generates an iterable to the input list, using iter() function. Then we use islice() function to split the original list into smaller tuples of 2 items each. We return an iterator to this tuple. Lastly, we call split_list() function inside a list to store all resultant tuples in a list.

Of course, if you want only a list of lists and not list of tuples, you can always loop through the iterable and convert each tuple into list.

for i in chunk(lst,2):
     print(list(i))

[1, 2]
[3, 4]
[5, 6]
[7, 8]
[9, 10]
[11]

4. Using Numpy

Numpy is another library that provides tons of useful functions for data analysis. In fact, it has a dedicated function array_split() that allows you to split a list into chunks. It just requires two arguments – the list to be split, and chunk size.

import numpy as np

lst = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
chunk_size=2
np.array_split(lst, chunk_size)

Please note, numpy is not present in default Python installation and needs to be installed separately using the following command.

$ pip install numpy

5. Using Loop

Lastly, we have the good old fashioned simple loop, where we iterate through the original list, and create the smaller lists of N size, by adding items to them one by one. This is perhaps the simplest solution of them all.

lst = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
chunk_size=2
for i in range(0, len(lst),chunk_size):
    print(lst[i:i+chunk_size])

# Output
[1, 2]
[3, 4]
[5, 6]
[7, 8]
[9, 10]
[11]

If you want, you can create a function using the above code. This will make it easy to reuse the above code.

def split_list(l, n):
    for i in range(0, len(l), n):
        print(lst[i:i+n])

split_list(lst, chunk_size)

Instead of printing each of the chunks, if you want to store them in a separate list, you can modify the function as shown below. Here we simply store the chunks in another list and return it at the end of for loop.

def split_list(l, n):
    new_lst=[]
    for i in range(0, len(l), n):
        new_lst.append(lst[i:i+n])
    return new_lst

x=split_list(lst, chunk_size)
print(x)

This method is useful for small-medium sized lists but not large lists since it loads the entire list as well as the chunks in memory.

Conclusion

In this article, we have learnt several different ways to split a list into N sized chunks. They all give the same result. Some solutions are more memory efficient than others while some are faster than the others. If you are working with large lists and have space constraints, then use yield keyword, since it is fast and does not require much memory to run. On the other hand, if you find it complicated, then use list comprehensions. If you are free to use libraries in your code, then you can try using itertools or numpy, both of which are really fast and efficient. If you are just looking for a very simple code to split a small-medium list into chunks, then you can use for loop as shown in the last solution, or use array_split() function offered by numpy library.

Also read:

How to Backup MySQL Database in Python
What Does Name-Main Idiom Do in Python
What Are Metaclasses in Python