Question

Why does the kernel need to validate pointers passed into a system call? What could go wrong if it did not validate user pointers?

Answer 1

The operating system provide certain services, now system calls provide interface to the services that are available by the operating system. the program can be execute into two modes:

1. User Mode - if program executed on user mode then that does not have direct access to the memory to the hardware and such resouces.
2. Kernel Mode - if program executed on kernal mode that program has direct access to the memory and hardware and such resources.

let us discuss on kernel mode to easy understand why kernel need validate pointers passed into system call.

if a program execute in kernel mode then it is privileged mode because it is having direct access to many of the resources but problem is that when program is executed in kernal mode, if the program happen to crash during its execution then the entire system would crash or entire system will come to a halt so that is one problem of kernel mode.

when program is executed in user mode, it may need to access of some of the resources like memory or hardware so when the program needs access to these resources it makes a call to your operating system telling that I need access to certain resources so it makes a call and when it makes a call for an instance the program switched from user mode to kernel mode so it can use those resources. the system calls made by program to access certain resources, the system call is programmatic way in which a computer request a service from the kernel of the operation system.

The kernel need to validate the pointers because attackers may can take benifits of reads from unchecked pointers. it is very imprtatance of validating pointers before passing into system call, if kernel forget to perform this check then possible of harm user get control of the system.

When you turn on your computer the CPU starts in level 0. the kernel then can configure stuff with the address through the instruction, then later drops privilege of the CPU to level, now hardware is kind of lockdown, we cannot configure register, hardware. we only can enter level 0 again, through a system call but we cannot control what will be executed, because the address is fixed.

For example, let us consider writing the program reading the data and copy the content from one file to another file.

now how system call are made for this simple task, firstly in order to copy the content of an input file to an output file we need to get the name of the input file, we need to know which is our input file from which we are going to copy the contents so the first step is to acquire the input file name so we need a system call in order to acquire the input file name and then we write a prompt to the screen that means you are displaying a prompt on the screen asking the user to enter the name of the input file so for writing the prompt to the scene you need another system call and then you need to accept the input that the user gives you so that need another system call. the system call are made when we want to access some resources of system.

Now, Acquiring the input file name we need system call, writing the prompt into the screen so going to user hardware output device therefore need an system call for accessing hardware and then accepting the input either the keyword or mouse which need system call. once we get the input file then we need to get the output file name from which the file going to copying the contents which is destination file name at his scenario we require system call.

When users enters the name of output file using the keyboard that need to accept the input for that we need an system call. we need to open the input file so start copy that things which need system call again, if file doesn't exists then you have to abort that means terminate the execution. we need to create the output file using the file entered, if the file exists then also you have to abort because you are try to create new file into which you are to copy the contents of the source file to the destination file. now the name of the output file that we provided or the name of distination provided if it already exists then you cannot create output file. you have to create new file are not allow to file already exists.

Now ready to write the input file to output file, for that need to run the loop because we need to copy all the contents of the input file to the output file. the loops runs unitl it fails, here also error can be accur like hardware errors at that time need to abort so we need system call. after everything copied from input file to output file then need to close the output file so closing output file again you need a system call. we need to display the message on screen copyring process is complete so let the user know that copyring process is complete.

Once done the process, need to terminate normally for that also we require system call. In this case we see that for many operation we require the system call.

according to example, it is easy to understand how kernel need to validate pointers pointers into system call, without validating what may problem accurs which execution.