Communication and Synchronization

Global-view model

reduction directive
bcast directive
barrier directive
shadow directive
reflect directive
gmove directive

Local-view model

Global-view model

reduction directive

The reduction construct performs a reduction operation among nodes.

An operation of reduction clause can use the following.

[F] +, *, -, .and., .or., .eqv., .neqv., max, min, iand, ior, ieor
[C] +, *, -, &, |, ^, &&, ||, max, min

bcast directive

The bcast construct performs broadcast communication from a specified node.

It specifies the transfer source with the from clause. When omitted, p[0] (p(1)) is assigned.
It specifies the transfer destination with the on clause. When omitted, the transfer is to all nodes.
The destination node set specified in the on clause must include the source node specified in the on clause.
The local variable a is broadcast to all nodes.
C
#pragma xmp bcast (a)
Fortran
!$xmp bcast (a)
[C] The local variable a held by node p[5] is transferred from node p[5] to node p[7].
```
#pragma xmp bcast (a) from p[5] on p[5:3]
```
[F] The local variable a held by node p(5) is transferred from node p(5) to node p(7).
```
!$xmp bcast (a) from p(5) on p(5:7)
```

barrier directive

The barrier construct specifies an explicit barrier at the point at which the construct appears.

If the on clause is omitted, barrier synchronization occurs for all execution nodes.

Example : all nodes wait until function func_a() ends.

func_a();
#pragma xmp barrier
func_b();

Fortran

call func_a()
!$xmp barrier
call func_b()

Example : only the nodes that have indices 10 through 20 of template t will be subject to barrier synchronization.
C
#pragma xmp barrier on t[10:11]
Fortran
!$xmp barrier on t(10:20)

shadow directive

The shadow directive allocates the shadow area for a distributed array.

reflect directive

The reflect construct assigns the value of a reflection source to the corresponding shadow object.

Example

#pragma xmp template t[8]
#pragma xmp nodes p[3]
int a[9], b[9];
#pragma xmp align a[i] with t[i]
#pragma xmp align b[i] with t[i]
#pragma xmp shadow a[1]

#pragma xmp loop on t[i]
for(i=0;i<9;i++){
   a[i] = init(i);     // Initialize a[]
}
#pragma xmp reflect (a)  // Synchronize

#pragma xmp loop on t[i]
for(i=1;i<8;i++){
   b[i] = a[i-1] + a[i] + a[i+1];
}

Fortran

!$xmp template t(9)
!$xmp nodes p(3)
integer :: a(9), b(9)
!$xmp align a(i) with t(i)
!$xmp align b(i) with t(i)
!$xmp shadow a(1)

!$xmp loop on t(i)
do i = 1, 9
   a(i) = init(i);     !  Initialize a()
end do
!$xmp reflect (a)  ! Synchronize

!$xmp loop on t(i)
do i=2, 7
   b(i) = a(i-1) + a(i) + a(i+1)
end do

In XMP/C, the shadow directive creates a shadow area at the upper and lower bounds of array a[]. The gray area is the reference element that is created.

The reflect directive synchronizes the shadow area. The directive generates communication between adjacent nodes.

When m-dimensions of n-dimensions array are distributed (n>m), shadow directive declares that no-distributed dimensions do not have sleeve area by inserting "0".

#pragma xmp template t[10]
#pragma xmp nodes p[*]
int a[10][20][30];
#pragma xmp align a[i][*][*] with t[i]
#pragma xmp shadow a[1][0][0]

Fortran

!$xmp template t(10)
!$xmp nodes p(*)
integer :: a(30,20,10)
!$xmp align a(*,*,i) with t(i)
!$xmp shadow a(0,0,1)

gmove directive

The gmove construct allows an assignment statement, which may cause communication, to be executed possibly in parallel by the executing nodes.

If neither in nor out is specified, the data referenced on the left and right sides must be located in the current node set. The data on the right side are transferred from the node containing that data and are received by and assigned to the node on the left side.

A distributed array between nodes, a local array or a local scalar variable is assigned.
The assignment statement is limited to simple assignment, without any arithmetic operations.
In XMP/C language, A[n:m] means that m elements from A[n].
In XMP/Fortran language, A(n:m) means that elements through n to m of array A.
If the right side is a local variable, the data must have the same value at all the nodes.
If the left side is a local variable, the same value is assigned (it is equivalent to a broadcast operation)
If both the right and left sides are distributed arrays, all-to-all communication is performed. If the left side is a duplicate array, this operation becomes a multicast.
If the right side is a distributed array, this becomes a broadcast communication from the specific node.

Example 1: Assignment statement 1: scalar variables

#pragma xmp gmove
s1 = s2;              // s1,s2 are scalar variables

Fortran

!$xmp gmove
s1 = s2              !  s1,s2 are scalar variables

Example 2: Assignment statement 2: using scalar variables

#pragma xmp gmove
a[3] = b[i][j];      // a,b are local arrays

Fortran

!$xmp gmove
a(3) = b(j,i)      !  a,b are local arrays

Example 3: Assignment statement 1: using distributed arrays

#pragma xmp gmove
a[:] = b[:];      // a,b are distributed arrays

Fortran

!$xmp gmove
a(:) = b(:)      ! a,b are distributed arrays

Example 4: Assignment statement 2: using distributed arrays

#pragma xmp gmove
a[1:9] = b[n:9]; Number of elements in both side must be the same

Fortran

!$xmp gmove
a(1:9) = b(n:n+8)   !  Number of elements in both side must be the same

Example 5: Assignment statement 3: using distributed arrays

#pragma xmp gmove
a[1:10] = c;    // c is a scalar variable. c is defined to a[1:10]

Fortran

!$xmp gmove
a(1:10) = c    !  c is a scalar variable. c is defined to a[1:10]

Example 6: Using the in clause

#pragma xmp nodes p[4]
int a[4], b[4];
#pragma xmp distribute t[block] onto p
// ..snip..
#pragma xmp task on p[1:2]
#pragma xmp gmove in
a[1:2] = b[2:2]

Fortran

!$xmp nodes p(4)
integer :: a(4), b(4)
!$xmp distribute t(block) onto p
! ..snip..
!$xmp task on p(2:3)
!$xmp gmove in
a(2:3) = b(3:4)

If the in clause is specified, then the assignment operation is performed after the node retaining the left side data acquires (gets) the corresponding right-side data by a remote copy operation. Correspondingly, the data referenced on the left side must be allocated to the current node set.

If the out clause is specified, then the assignment operation is performed after the node retaining the right-side data updates (puts) the corresponding left-side data by a remote copy operation. Correspondingly, the data referenced on the right side must be allocated to the current node set.

Local-view model

A one-sided communication program can be coded as follows.

int a[100]:[*], b[100];
if(xmpc_this_image() == 0){
  a[0:100]:[1] = b[0:100];  // PUT
}

Fortran

integer :: a(100)[*], b(100)
if (this_image() == 1) then
  a(1:100):[2] = b(1:100)  ! PUT
end if