How to Execute System Command in Python

Sometimes software developers need to run program or execute system command in Python. This is commonly required to perform background tasks and critical processes. For example, they may want to run shell script from Python program. There are several ways to do this in Python. In this article, we will learn how to run shell commands in Python.

Why Run Shell Commands in Python

There are several reasons why developers need to run system commands and programs from within Python.

Often software developers need to run critical commands or background processes along with their applications/websites. They may be OS related commands or file manipulations. They need to do this without disturbing the execution thread of their main Python program. That is why it is super useful to be able to run shell commands in Python. For example, it is lot easier to run critical backups and system maintenance commands this way instead of using Python modules for this purpose.

Another reason is that there are some things that are are easily done using system commands. For example, file manipulation is a lot easier using shell commands such as mv, cp, etc. instead of doing it using Python script.

Sometimes, you may also need to run one Python script from within another Python script. In this case also, you can use this solution.

Lastly, it saves a lot of coding, since you already have many system commands and programs at your disposal. You just need to call them from your Python script instead of coding them from scratch.

Python provides two main modules, subprocess and os, for this purpose. Each of them provide multiple functions to run system commands and programs. Let us learn a little bit about these two modules.

What is OS module

Python provides OS module that provides many useful functions & constants to interact with underlying operating system. You can use the same commands to work with different operating systems. It allows you to create & modify files & directories. You can also use it to manage input, output, processes and environment variables.

What is Subprocess

Subprocess is a helper module built to replace several functions of OS module. But over time, it has evolved into a robust and comprehensive library. It also allows you to create files, directories and processes, work with their input/output/errors. Its functions are a little more complicated than those provided by OS module. But its main advantage is that subprocess functions provide better control over system commands by returning a handle, as well as error codes.

How to Execute Program or System Command in Python

Let us look at the key functions provided by subprocess and os modules, that can be used to run system commands from Python.

1. Using subprocess.Popen()

subprocess.Popen() is a popular upgrade to os.popen() command (below) that is more comprehensive and versatile. It has a slightly complicated syntax but handles most use cases easily. You can pass the command as a string, or a list. Here is an example to pass command as a string.

import subprocess

subprocess.Popen('ls -all')

Here is an example where the command is passed as a list. In this case, you do not need to worry about escaping.

import subprocess
subprocess.popen(['ls','-l'])

In above cases, subprocess will spawn a separate process to run the command. If you want to run the process in shell itself, you can pass shell=True argument.

import subprocess
subprocess.popen(['ls','-l'], shell=True)

OR 

import subprocess
subprocess.popen('ls -all', shell=True)

2. Using subprocess.run()

If you run Python 3.5+, you can also use subprocess.run() function to execute system command in Python. Its syntax is similar to that of subprocess.Popen(). It requires you to pass the command as a list of strings, and not a string. It is more flexible than using subprocess.Popen() as it returns a process object on completion of command. It also supports shell=True argument like subprocess.Popen().

import subprocess
subprocess.run(["ls", "-a"], shell=True)

3. Using subprocess.call()

subprocess.call() is an old function available on python <= 3.4. In fact, on older Python versions you will not find subprocess.run() and you will need to use this function instead. Its syntax is similar to that of system.run() function. You need to pass the command, along with its arguments, as a list of strings. It also allows you to run the command in a shell using shell=True option. Here is an example to run ls command using system.call().

import subprocess
subprocess.run(["ls", "-a"], shell=True)

4. Using subprocess.check_output()

Often software developers need to not only run system commands but also store their outputs for later use. You can store the output of your command using subprocess.check_output() function. Here is an example to illustrate it.

import subprocess
out = subprocess.check_output('date')
print('Current date is:', out.decode("utf-8"))

subprocess.check_output() returns the output as byte string. You need to call decode() function to convert it into UTF-8 string. Here is the output of above code.

('Current date is:', u'Wed Nov 20 06:23:30 UTC 2024\n')

5. Using os.system()

OS module offers system() command that is one of the oldest and reliable ways to run system commands in Python. Here is its syntax.

os.system(command)

You can use it to pass arguments as well as run multiple commands, along with them input and output redirection. Here is are some examples to illustrate its use.

import os

os.system('pwd')
os.system("ls -all | grep 'test'")

Please note, in the last command above, we enclose the substring to be searched within single quotes, whereas the actual command within double quotes, to avoid conflict. You will need to escape special characters while using them in os.system().

Os.sytem() does not return any file handler that you can use to control the process. Once you call this function, it will start running the command.

6. Using os.popen()

Os.popen() is another function that you can use to run programs and scripts from within Python. It is more flexible than using os.system() since it gives you a file object to handle its input, output and error. Here is its syntax.

os.popen(command, mode, buf_size)

You can pass command, including arguments, as a string. Here are examples to demonstrate it.

import os
os.popen('ls -l').read()

In the above case, the command string is directly sent to shell. So you may need to escape the special characters in it.

Here is an example to save the file object returns by os.popen() and use it to control the process.

import os

f = os.popen('ls -a').read()
f.close()

Conclusion

In this article, we have learnt several different ways to easily run programs and system command in Python. You can use any of these methods as per your requirement. Using subprocess module offers more control and flexibility than using OS module.

Also read:

How to Check If File Exists in Python
How to Merge Two Dictionaries in Python
How to Split Python List Into Even Chunks