Question

What is the main benefit of thread synchronization? Why do you think CUDA does not allow block synchronization?

Answer 1

Thread Synchronization is the process of excecuting more than one threads that share the same critical resources at the same time (i.e. concurrently) in such a way that no resource conflicts happen. This is generally achieved by restricting the execution of a thread until another thread or part of it is not executed completely. This simplest example in this scenario is:

Consider two threads that share the same critical resources ( such resources which can only be used by a single process at a time). If these threads are allowed to run asynchronously (i.e. without synchronization) then they can try to access the same critical resource at the same time (as actual thread execution in CPU depends on the time slices provided by the OS and is effectively unpredictable). So, this can cause several anomalies like unpredictable and inconsistant outputs resulting in serious bugs or even system crashes or a condition called Race Condition.

So, the main benefits of thread synchronization are:

1. Prevention of Race Condition.
2. Prevention of inconsitant or unexpected output

CUDA does not allow block synchronization because:

1. Block synchronization is impractical if large and demanding processes are considered. Consider a process that requires a millions blocks to execute but the GPU running it only allows 1,000 blocks at a time. In this case, block synchronization can easily become obsolete. These cases are also very frequent when programs like Machine Learning are considered.
2. The context saving required to achieve block synchronization is very huge and hence very slow.
3. Alternatives to block syncronization are very fast and efficient like kernel switching/ launching a new kernel.