“execvp” system call in Python - Everything you need to know!
The exec family of the system calls is used to run a command or a code file in a new process. In Linux, this new process is created by replacing the process that made the exec call, but in Windows, things are a bit different. In this blog, we will be covering the execvp system call in Python. We will also be solving an interesting problem using the stuff learned throughout this blog!
Problem statement and the motivation
I was recently assigned an assignment for Operating Systems, which had an interesting question dealing with the execvp system call. Now, there are some implementations for this question available on the internet, but all of them are written in C. In pursuing a pythonic implementation, I went on a journey of exploration, and I will use it here as motivation! We will be applying the theoretical knowledge of execvp to solve this particular question at the end of this blog! The question -
Write a collection of programs p1, p2 and p3 such that they execute sequentially with the same PID and each program should also print its PID. The user should be able to invoke any combination of these programs to achieve the required functionality. For example - Consider three programs twice, half and square which accept only one integer as argument and does some specific operation. These operations may be like -
$ twice 10 prints 20 and some number which is its PID
$ half 10 prints 5 and some number which is its PID
$ square 10 prints 100 and some number which is its PID
Now the user should be able to combine these programs in any combination to achieve the desired result.
For example -
$ twice square half twice half 10
should calculate half(twice(half(square(twice(10))))) and print 200 as result. It should also print the process ids of each program as it executes. Note that the process-id printed by each of these programs should be the same, in this case.
$ square twice 2
should calculate twice(square(2)) and print 8 as result, and the process id of square and twice, which should be the same. The evaluation order is from left to right
Note that the last argument is integer, and the remaining arguments are the programs to be invoked.
This should be generally applicable to any n number of processes, all of which are written by you.
Documentation
Let us start by going through the documentation of execvp -
os.execvp(file, args)
These functions all execute a new program, replacing the current process; they do not return. On Unix, the new executable is loaded into the current process, and will have the same process id as the caller. Errors will be reported as OSError exceptions.
Key takeaways -
- All the
execcalls are implemented in theoslibrary. -
os.execvptakes in 2 arguments, file and args. - The new process is not a child process but rather the new process substitutes the current process.
- No return value.
- On
UNIXsystems, the process ID (PID) remains the same as the caller. - Saving you some time, I discovered that the file is supposed to be an executable file and the args are supposed to have the executable file name too. We will look into this in more detail below!
A minimal example
Now that we know some stuff about execvp, let us try using it in our code. The tradition is to start with a “Hello world” program, and we cannot go against the tradition. Further, we will be needing 2 Python files, a caller, and a file that prints “Hello world” -
caller.py
1
2
3
4
import os
command = ["python", "hello.py"]
`os.execvp`(command[0], command)
hello.py
1
print("Hello world")
Notice how execvp in caller.py takes in the name of an executable file (python) as the first argument and the complete command (including the executable file’s name) as the second argument. Running caller.py gives us “Hello world”, everything as expected!
Internals of execvp and playing with process IDs
According to the documentation, execvp should replace the caller process with the new process instead of creating a subprocess in UNIX systems. This means that the process ID should not change when the system call is made. Let us try this out, but first, let us create 3 different files for 3 different functions — square, half, and double —
1
2
3
4
5
6
7
8
9
10
import os
def square(num):
print("SQUARE PID:", os.getpid(), "| RESULT:", num ** 2)
return num ** 2
if __name__ == "__main__":
square(10)
1
2
3
4
5
6
7
8
9
10
import os
def half(num):
print("HALF PID:", os.getpid(), "| RESULT:", num / 2)
return num / 2
if __name__ == "__main__":
half(10)
1
2
3
4
5
6
7
8
9
10
import os
def double(num):
print("DOUBLE PID:", os.getpid(), "| RESULT:", num * 2)
return num * 2
if __name__ == "__main__":
double(10)
The only new thing in these files is os.getpid(), which would return the process ID of the process running these files. Let us also modify our caller to execute one of these files —
1
2
3
4
5
6
7
import os
import sys
print("CALLER PID:", os.getpid())
command = [sys.executable, "double.py"]
`os.execvp`(command[0], command)
We now check the process ID in our caller too. Additionally, to avoid making a mistake in the executable file’s name, we will be using sys.executable now! Executing caller.py in Windows gives us the following output -
CALLER PID: 3156
DOUBLE PID: 20940 | RESULT: 20The results are definitely weird as the process IDs are not the same. Let us try the same code in WSL (Windows Subsystem for Linux) -
CALLER PID: 34
DOUBLE PID: 34 | RESULT: 20The results match with the documentation! As the process IDs are the same, the caller process must’ve been replaced with a new process through the execvp system call. Note that we did not have to change the executable file’s name from python to python3 as sys.executable automatically picked it up!
Thus, os.execvp behaves differently on Windows and on UNIX systems. On Windows, it spawns or creates a new process (a child or a subprocess) whereas, on a UNIX system, it replaces the original process with a new process!
Developing a CLI
Now that we know how to use execvp in Python let us move on to our original question! Let us create a file to control everything through the command line —
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
import os
import sys
import argparse
import subprocess
from half import half
from double import double
from square import square
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument(
"-l", "--list", nargs="+", help="<Required> Set flag", required=True
)
args = parser.parse_args()
print("CLI PID:", os.getpid())
print(args.list)
print()
init_val = args.list[-1]
args.list = [file + ".py" for file in args.list[:-1]]
args.list.append(init_val)
for i in range(len(args.list) - 1):
if args.list[i] == "half.py":
cmd = [sys.executable] + args.list
`os.execvp`(cmd[0], cmd)
elif args.list[i] == "double.py":
cmd = [sys.executable] + args.list
`os.execvp`(cmd[0], cmd)
elif args.list[i] == "square.py":
cmd = [sys.executable] + args.list
`os.execvp`(cmd[0], cmd)
This file, when executed, accepts a list of CLI arguments which are the function names and a number. The last CLI arg must be a number that has to be processed through various operations. A usage example for this file would be —
python cli.py -l twice square half twice half 10which should internally translate to —
half(twice(half(square(twice(10)))))and should print all the results and relevant process IDs.
Remember, os.execvp does not return anything. Hence, once a new python file is called for execution, the flow of control won’t come back to our CLI file. To tackle this we must add additional code in our operation functions which would call the next file without returning back to cli.py.
Modifying the operation files
The code in every operation file should be modified by adding —
1
2
3
4
5
6
7
8
9
10
11
12
if __name__ == "__main__":
n = operation(float(sys.argv[-1]))
sys.argv[-1] = str(n)
if sys.argv[1] not in ["double.py", "half.py", "square.py"]:
print()
print("FINAL PID:", os.getpid(), "| FINAL RESULT:", sys.argv[1])
sys.exit(0)
cmd = [sys.executable] + sys.argv[1:]
`os.execvp`(cmd[0], cmd)
where operation is either double, half, or square.
The code takes in the last CLI argument using sys.argv[-1] and passes it into the relevant operation function. This last argument is then replaced with the obtained result, and all the CLI args except the first one are passed into os.execvp, which calls the next operation file!
In between, we also need to add a condition to exit if we have reached the last argument, which is a number. This last argument would be the final result as we would have processed all the arguments (function names) before that!
Let us modify every operation file —
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
import os
import sys
def double(num):
print("DOUBLE PID:", os.getpid(), "| RESULT:", num * 2)
return num * 2
if __name__ == "__main__":
n = double(float(sys.argv[-1]))
sys.argv[-1] = str(n)
if sys.argv[1] not in ["double.py", "half.py", "square.py"]:
print()
print("FINAL PID:", os.getpid(), "| FINAL RESULT:", sys.argv[1])
sys.exit(0)
cmd = [sys.executable] + sys.argv[1:]
`os.execvp`(cmd[0], cmd)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
import os
import sys
def square(num):
print("SQUARE PID:", os.getpid(), "| RESULT:", num ** 2)
return num ** 2
if __name__ == "__main__":
n = square(float(sys.argv[-1]))
sys.argv[-1] = str(n)
if sys.argv[1] not in ["double.py", "half.py", "square.py"]:
print()
print("FINAL PID:", os.getpid(), "| FINAL RESULT:", sys.argv[1])
sys.exit(0)
cmd = [sys.executable] + sys.argv[1:]
`os.execvp`(cmd[0], cmd)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
import os
import sys
def half(num):
print("HALF PID:", os.getpid(), "| RESULT:", num / 2)
return num / 2
if __name__ == "__main__":
n = half(float(sys.argv[-1]))
sys.argv[-1] = str(n)
if sys.argv[1] not in ["double.py", "half.py", "square.py"]:
print()
print("FINAL PID:", os.getpid(), "| FINAL RESULT:", sys.argv[1])
sys.exit(0)
cmd = [sys.executable] + sys.argv[1:]
`os.execvp`(cmd[0], cmd)
These files will now call each other repeatedly and the whole system would work without returning to a previous file!
Final results
Running the following —
python cli.py -l double square half double half 10on Windows results in —
CLI PID: 7376
['double', 'square', 'half', 'double', 'half', '10']
DOUBLE PID: 32496 | RESULT: 20.0
SQUARE PID: 15144 | RESULT: 400.0
HALF PID: 4860 | RESULT: 200.0
DOUBLE PID: 24928 | RESULT: 400.0
HALF PID: 27868 | RESULT: 200.0
FINAL PID: 27868 | FINAL RESULT: 200.0IT WORKS! The program gives us the desired output! Notice how the PIDs are not the same, something that was discussed in great detail above.
Running the following —
python3 cli.py -l double square half double half 10on Windows Subsystem for Linux results in —
CLI PID: 33
['double', 'square', 'half', 'double', 'half', '10']
DOUBLE PID: 33 | RESULT: 20.0
SQUARE PID: 33 | RESULT: 400.0
HALF PID: 33 | RESULT: 200.0
DOUBLE PID: 33 | RESULT: 400.0
HALF PID: 33 | RESULT: 200.0
FINAL PID: 33 | FINAL RESULT: 200.0THIS WORKS TOO! The program gives us the desired output again! This time the PIDs stay the same as no subprocesses were created!
Summary
In the above blog, we understood how to use a system call belonging to the exec family of system calls. We further saw how the behavior of execvp differs in Windows and UNIX systems. In the end, we solved a question that had no Pythonic solution on the internet! :)