A common single-processor "trick" for coordinated access to a shared memory region is to use FIFO scheduling between two threads running at the same priority. The idea is that one thread will access the region and then call SchedYield() to give up its use of the processor. Then, the second thread would run and access the region. When it was done, the second thread too would call SchedYield(), and the first thread would run again. Since there's only one processor, both threads would cooperatively share that processor.
This FIFO trick won't work on an SMP system, because both threads may run simultaneously on different processors. You'll have to use the more "proper" thread synchronization primitives (e.g., a mutex), or use BMP to tie the threads to specific CPUs.