How to use Python subprocess to run OS commands.
I like using
os.system because you can run your favorite Linux commands. The
os module has built-in methods, but often they aren’t mapped exactly to Linux’s commands.
For example, show the current working in directory.
import os os.getcwd()
Then usually it’s easier to remember this:
import os os.system("pwd")
I’ve seen that
os.system doesn’t wait to complete, similar to what you would expect from a callback function.
for url in urls: os.system("curl some-resource") print("Hello")
Not exactly with this example, but I’ve seen the program continue execution before
os.system is completed, which leads to errors if the program depends on the output of
Replace os.system with subprocess.call
subprocess.call waits for the command to complete.
import os os.system("curl some-resource") # it might continue executing before os.system is done
import subprocess subprocess.call("curl some-resource", shell=True) # it waits for subprocess to complete
Use sleep() to wait for the process to start
I have seen this problem as well:
subprocess.call("do something here") run_this_function()
run_this_function() executes first and then
run_this_function() which uses a file produced by the subprocess, and it shows this error:
some-file.csv not found
Then it runs the process that was supposed to produce this file.
subprocess.call supposed to wait until it’s done to execute the next line? Sure. According to the Python docs, it waits to complete. But the docs don’t say anything about how long does it take for a process to start. Or maybe it starts, returning zero, executes the next line, and then it completes.
Quick fix with
subprocess.call("do something here") sleep(5) run_this_function()
I haven’t timed how long does it take for
subprocess.call to start and end. Another quick check was testing if the output is produced before running the next line. This could be done in a unit test. Or like this:
subprocess.call("do something and create some-file.csv") sleep(5) if 'some-file.csv' in os.listdir(): run_this_function()