Ada-2005




Research

A great deal of the research on real-time scheduling is slowed down from being used in actual applications by the lack of adequate implementations that provide the necessary abstractions. Ada has evolved over the past decade to offer a comprehensive set of mechanisms that can deliver modern real-time scheduling theory to the systems engineer.

Some current projects:

Below we show how Ada-2005 provides the necessary abstractions to construct the Multiprocessor Priority Ceiling Protocol (MPCP) proposed by Sha, Rajkumar, and Lehoczky. Additionally, we show the Stack Resource Policy (SRP) introduced by Baker in 1990. Furthermore, we provide a merge sort program for a demo on synchronization and resource sharing.

The goal of MPCP is to bound the remote blocking duration of a task. For example, it is wrong for a task to be blocked on one processor while another task executes on a different processor outside a critical section.

Consider, a situation where three periodic tasks J1, J2, J3 are bound to processor P1, and another task J4 is bound to processor P2. Let J1 have the shortest period, and let J4 have the longest period. Thus, J1 has the highest priority.

Example 1


J4 is bound to processor P2. If J4 attempts to lock semaphore S it will be blocked, since S is currently locked by J3 executing on processor P1. However, task J1 now pre-empts J3 on processor P2, J4 will be blocked until J1 completes. Thus, the blocking time of J4 will continue until arriving higher priority tasks on P1 (such as J1 and J2) complete execution or suspend themselves. Therefore, we prevent a task to be blocked on one processor, while another task executes on another processor outside a globally shared resource.

In order to prevent deadlocks, MPCP must keep a record of the state of each resource. Using this information, it can determine if the allocation of a particular resource to a given task would cause a deadlock situation. MPCP implements deadlocks avoidance by assigning a priority ceiling (PC) to every resource in the system. The PC of resource R is defined as the priority of the highest priority task that uses R. For example, consider the example below: Here since J4 has a lock on S1, J1 is prevented for locking S2 since both S1 and S2 have the priority of J1.

Example 2


With MSRP, it is an extra step than MPCP--a task can only become ready for execution if its preemption level is higher than the preemption levels of the resources currently locked. For example, consider the example below: Here since J2 has locked S, then if J1 arrives, it cannot even start until J2 has released S.

Example 3



MPCP in Ada-2005

Ada-2005 provides many features for implementing MPCP and MSRP efficiently.
  • Ada-2005 provides the ability to access resources according to priority. The predetined queuing policy Priority-Queuing ensures the servicing of entry queues based on priorities.


  • Ada-2005 provides the ability to identify tasks. The package Ada.Task-Identification is used for this purpose. The package includes the definitions of the data type Task-ID and the function Current-Task that returns a value to identify the calling task.


  • Ada-2005 provides the ability to determine the current active priority of a task and the ability to set the active priority of a task dynamically. In addition, Ada-2005 provides the ability to set the priority of protected objects dynamically. This is provided in the package Ada.Dynamic-Priorities by the functions Get-Priority and Set-Priority.


  • Ada-2005 provides the ability to release a resource to wait for a certain condition to occur. In MPCP and MSRP, if a task T calls a Request operation, the scheduler needs to try to service the request. If the locking condition is not satisfied, it is necessary for task T to release the scheduler to process other requests and to wait for the condition.

Additional Experimental Results

We validated the accuracy of our implementations with a variety of test inputs. To compare MPCP and MSRP, we tested the MPCP and MSRP implementations with gnat-gpl on an x86_64 Linux system. It has a tightly coupled, shared memory multi-processor architecture and hosts a version of GNU/Linux 2.6.9. The system has 8 processors, 32 GB of RAM, and four 1 GB DIMMS directly managed by each processor. In one of the experiments, we focused on the total lock-waiting time. This is the total time taken by the resource server (scheduler) to grant a lock. The total time includes: (1) Any time spent in the queue (2) Checking if the lock rules are met and (3) Raising the priority of the task to the level of the lock, if necessary.

Protocol Average lock waiting time (light workloads) Average lock waiting time (heavy workloads)
MPCP 0.062566(secs) 0.114257(secs)
MSRP 0.063962(secs) 0.143038(secs)


figure1
Figure 1: Percentage of deadline misses -- task sets with randomly selected periods / heavy resource usage.


figure2
Figure 2: Percentage of deadline misses -- task sets with randomly selected periods, and with lighter resource usage


figure3
Figure 3: Percentage of priority changes ratio by the MPCP, the MSRP, and the FCFS.


figure4
Figure 4: Percentage of context switches by the MPCP, the MSRP, and the FCFS.


figure4
Figure 5: Lock waiting ratio -- task sets with randomly selected periods

Run-time Comments

  • There are all sorts of interesting things to see here. The two algorithms ran in roughly the same amount of time. Both codes are so similar that I don't think the average user would see a difference.
  • MPCP ran slightly faster with heavy workloads (the shared resource length was longer).
  • It is possible to get MSRP to beat MPCP code with lighter workloads.

Conclusion

Starting with the purpose of supporting SMP systems, we have shown that it is possible to support applications that want to run on multiprocessor systems with Ada. We have shown different methods of how programmers could map tasks to processors. Moreover, we have shown that Ada provides a strong degree of support and protection to critical resources. Apart from this, we were successful in showing that MSRP is a refinement of MPCP, and in most respects, MSRP is a consistent extension of MPCP. There is no support for MPCP in the POSIX standard. If we attempt to implement it on top of threads, we would need a user-level way to keep track of the current system priority, which is a global quantity. We have shown that both MPCP and MSRP have strengths and weaknesses. The main advantage of MPCP over MSRP is that it does not raise priority until that is proven necessary in preventing a priority inversion. So in many cases, the priority need not change. For this reason, MPCP adheres to priority scheduling better. However, there is still an overhead to determine whether a priority change is needed. The gain is that, there are fewer cases of priority inversion. With MSRP, whenever we raise the priority of the lock holder that is causing a temporary priority inversion, we gain over ‘no protocol’ by the fact that this inversion is strictly bounded in duration. It happens less often with MPCP, which is a gain on the average. On the other hand, we have shown that in the worst-case, the number of context-switches is lower with MSRP than MPCP.


An Ada-2005 implementation of MPCP is shown below

-- Multiprocessor Priority Ceiling Protocol (MPCP)     

  -- Raised Exceptions when the basic assumptions of MPCP are violated
  Release_Error : Exception;
  Ceiling_Error : Exception;
  Nesting_Error : Exception;

  Locked    : constant Boolean := True;
  Nesting   : constant Boolean := True;

  -- application dependent
  -- Range of priorities of tasks accessing critical sections
  TASK_MIN_Priority  : constant Priority     := -- application dependent. 	
  TASK_MAX_Priority  : constant Priority     := -- application dependent.

  -- application dependent 
  Scheduler_Priority : constant Any_Priority := -- application dependent.;	

  subtype Task_Prange is Any_Priority range TASK_MIN_Priority..TASK_MAX_Priority;
  type Task_Priority is array (Positive range <>) of Task_Prange;

  type intArray is array (Positive range <>) of Integer;
  type Resource_Count  is array (Positive range <>) of Boolean; 
  type Resource_Locking_Task is array (Positive range <>) of Task_ID; 
  type Resource_Priority is array (Positive range <>) of Task_Prange;

  type Scheduler is protected interface;

    procedure Request 
      (Object: in out Scheduler; 
       tid: Task_ID; 
       S: Positive;
       P: Positive;
       T: Time) is abstract;

    procedure Release 
      (Object: in out Scheduler; 
       tid: Task_ID; 
       S: Positive) is abstract;

  protected type Resource_Scheduler (N: Positive; Scheduler_Priority : Any_Priority) is new Scheduler with

    -- Resource_Scheduler must have a Priority higher than all calling tasks
    pragma Priority(Scheduler_Priority);

    overriding
    procedure Release (tid: Task_ID; S: Positive);

    overriding
    entry Request
       (tid: Task_ID; S: Positive; P: Positive; T: Time);
 
  private 
    Resource_Status            : Resource_Count(1..N);
    Locking_Task  	       : Resource_Locking_Task(1..N);
    Old_Priority               : Task_Priority(1..N);
    Active_Priority            : Task_Priority(1..N);
    Ceiling_Level              : Resource_Priority(1..N);
    Ceiling_PH                 : Resource_Priority(1..N);
    Task_P      	       : intArray(1..N);
    --resource processor
    Resource_P      	       : intArray(1..N);
    --resource type for DPCP
    DPCP_T      	       : intArray(1..N);
    --resource value for DPCP
    DPCP_V      	       : intArray(1..N);

    -- if True, then Gate1 Ready, else Gate2 Ready
    Current_Queue      	       : Boolean;
    Ready_Queue        	       : Boolean := True;

    -- if True, then Gate1 Ready, else Gate2 Ready
    Queue_Ready              : Boolean := True;  
    Gate1                    : Boolean;
    Gate2                    : Boolean;

    entry Queue1
      (tid: Task_ID; S: Positive; P: Positive; T: Time);

    entry Queue2
      (tid: Task_ID; S: Positive; P: Positive; T: Time);

    function Check_Ceiling_Level
       (tid: Task_ID; Task_Priority: Task_Prange) return Boolean;

    function Check_Nesting
       (tid: Task_ID; S: Positive) return Boolean;

  end Resource_Scheduler;
  protected body Resource_Scheduler is

    procedure Release (tid: Task_ID; S: Positive) is
    begin
      if(tid /= Locking_task(S)) then
        raise Release_Error;
      else
        Set_Priority(Old_Priority(S),tid);
        Resource_Status(S) := False;
        DPCP_V(DPCP_T(S))  := DPCP_V(DPCP_T(S)) + 1;
        Locking_Task(S)    := Null_Task_ID;
	-- if current queue is empty, then switch queues
        if Queue_Ready then 
          if Queue1'Count > 0 then
            Gate1 := TRUE;
          end if;  
        else 		
          if Queue2'Count > 0 then
            Gate2:= TRUE;
          end if;
        end if;               
      end if;
    end Release;

    entry Request
       (tid: Task_ID; S: Positive;  P: Positive; T: Time) when (N>0) is
           Task_Priority: Task_Prange := Get_Priority(tid);
    begin

      if (Task_Priority > Ceiling_Level(S)) then 
		raise Ceiling_Error;

      elsif (Check_Nesting(tid, S) = False) then 
		raise Nesting_Error;

      elsif (Resource_Status(S) = not Locked) and then
            (Check_Ceiling_Level(tid, Task_Priority)) and then
            (DPCP_V(DPCP_T(S)) > 1) then

          Resource_Status(S)  	:= Locked;
          Locking_Task(S) 	:= tid;
	  Task_P(S) 		:= P;
          Old_Priority(S) 	:= Task_Priority;
          DPCP_V(DPCP_T(S))	:= DPCP_V(DPCP_T(S)) - 1;

          if(Resource_P(S) /= P) and then 
            (Task_Priority < Ceiling_PH(S)) then
            Active_Priority(S) := Ceiling_PH(S);
            Set_Priority(Ceiling_PH(S), tid);         
	  else
            Active_Priority(S) := Task_Priority;
          end if;
      else

          if(Resource_Status(S) = Locked) and then
	    (Active_Priority(S) < Task_Priority) then
              Active_Priority(S) := Task_Priority;
              Set_Priority(Task_Priority, Locking_task(S));
          end if;
	   -- Resource is locked, try again later
          if Queue_Ready then 	
            requeue Queue1;
          else 			
            requeue Queue2;
          end if;
      end if;
    end Request;

    function Check_Ceiling_Level
       (tid: Task_ID; Task_Priority: Task_Prange) return Boolean is
    begin
      for I in 1 .. N loop
        if (Resource_Status(I) = Locked) and then
           (Locking_Task(I) /= tid) and then
             (Ceiling_Level(I) >= Task_Priority) then  
		return False;
        end if;
      end loop;
      return True;
    end Check_Ceiling_Level;

    function Check_Nesting
       (tid: Task_ID; S: Positive) return Boolean is
    begin
      for I in 1 .. N loop
        if  (Resource_Status(I) = True) and then
            (Locking_Task(I) = tid) then
		return False;
        end if;
      end loop;
      return True;
    end Check_Nesting;

    entry Queue1
      (tid: Task_ID; S: Positive; P: Positive; T: Time) when Gate1 is
        Task_Priority : Task_Prange := Ada.Dynamic_Priorities.Get_Priority(tid);
    begin 
      ---------------------------------------------------------------------------
      if Queue1'Count = 0 then
        Gate1      := False;
        Queue_Ready := False;
      end if; 

      if (Resource_Status(S) = not Locked) and then 
            (Check_Ceiling_Level(tid, Task_Priority)) and then
            (DPCP_V(DPCP_T(S)) > 1) then

          -- Resource is available, then lock it
          Resource_Status(S)  	:= Locked;
          Locking_Task(S) 	:= tid;
	  Task_P(S) 		:= P;
          Old_Priority(S) 	:= Task_Priority;
          DPCP_V(DPCP_T(S))	:= DPCP_V(DPCP_T(S)) - 1;

          if(Resource_P(S) /= P) then 
            Active_Priority(S) := Ceiling_PH(S);
            Set_Priority(Ceiling_PH(S), tid);
	  else
            Active_Priority(S) := Task_Priority;
          end if;
      else
          -- resource is still not available, then requeue again
	  requeue Queue2;
      end if;
      ---------------------------------------------------------------------------
    end Queue1; 

    entry Queue2
      (tid: Task_ID; S: Positive; P: Positive; T: Time) when Gate2 is
        Task_Priority  : Task_Prange := Ada.Dynamic_Priorities.Get_Priority(tid);
    begin 
      ---------------------------------------------------------------------------
      if Queue2'Count = 0 then
        Gate2       := False;
        Queue_Ready := True;
      end if; 

      if (Resource_Status(S) = not Locked) and then 
            (Check_Ceiling_Level(tid, Task_Priority)) and then
            (DPCP_V(DPCP_T(S)) > 1) then

          -- Resource is available, then lock it
          Resource_Status(S)  	:= Locked;
          Locking_Task(S) 	:= tid;
	  Task_P(S) 		:= P;
          Old_Priority(S) 	:= Task_Priority;
          DPCP_V(DPCP_T(S))	:= DPCP_V(DPCP_T(S)) - 1;

          if(Resource_P(S) /= P) then 
            Active_Priority(S) := Ceiling_PH(S);
            Set_Priority(Ceiling_PH(S), tid);
	  else
            Active_Priority(S) := Task_Priority;
          end if;
      else
          -- resource is still not available, then requeue again
	  requeue Queue1;
      end if;
      ---------------------------------------------------------------------------
    end Queue2; 

  end Resource_Scheduler;

An Ada-2005 implementation of MSRP is shown below

  -- Raised Exceptions when the basic assumptions of MPCP are violated
  Release_Error : Exception;
  Ceiling_Error : Exception;
  Nesting_Error : Exception;

  Locked    : constant Boolean := True;
  Nesting   : constant Boolean := True;

  -- application dependent
  -- Range of priorities of tasks accessing critical sections
  TASK_MIN_Priority  : constant Priority     := -- application dependent. 	
  TASK_MAX_Priority  : constant Priority     := -- application dependent.

  -- application dependent 
  Scheduler_Priority : constant Any_Priority := -- application dependent.	

  subtype Task_Prange is Any_Priority range TASK_MIN_Priority..TASK_MAX_Priority;
  type Task_Priority is array (Positive range <>) of Task_Prange;

  type intArray is array (Positive range <>) of Integer;
  type Resource_Count  is array (Positive range <>) of Boolean; 
  type Resource_Locking_Task is array (Positive range <>) of Task_ID; 
  type Resource_Priority is array (Positive range <>) of Task_Prange;

  type Scheduler is protected interface;

    procedure Request 
      (Object: in out Scheduler; 
       tid: Task_ID; 
       S: Positive;
       P: Positive;
       T: Time) is abstract;

    procedure Release 
      (Object: in out Scheduler; 
       tid: Task_ID; 
       S: Positive) is abstract;

  protected type Resource_Scheduler (N: Positive; Scheduler_Priority : Any_Priority) is new Scheduler with

    -- Resource_Scheduler must have a Priority higher than all calling tasks
    pragma Priority(Scheduler_Priority);

    overriding
    procedure Release (tid: Task_ID; S: Positive);

    overriding
    entry Request
       (tid: Task_ID; S: Positive; P: Positive; T: Time);
 
  private 
    Resource_Status            : Resource_Count(1..N);
    Locking_Task  	       : Resource_Locking_Task(1..N);
    Preemption_Level           : Task_Priority(1..N);
    Active_Priority            : Task_Priority(1..N);
    Ceiling_Priority           : Resource_Priority(1..N);
    Ceiling_PH                 : Resource_Priority(1..N);
    Task_P      	       : intArray(1..N);
    --resource processor
    Resource_P      	       : intArray(1..N);
    --resource type for DPCP
    DPCP_T      	       : intArray(1..N);
    --resource value for DPCP
    DPCP_V      	       : intArray(1..N);

    -- if True, then Gate1 Ready, else Gate2 Ready
    Current_Queue      	       : Boolean;
    Ready_Queue        	       : Boolean := True;

    -- if True, then Gate1 Ready, else Gate2 Ready
    Queue_Ready              : Boolean := True;  
    Gate1                    : Boolean;
    Gate2                    : Boolean;

    entry Queue1
      (tid: Task_ID; S: Positive; P: Positive; T: Time);

    entry Queue2
      (tid: Task_ID; S: Positive; P: Positive; T: Time);

    function Check_Ceiling
       (tid: Task_ID; Task_Priority: Task_Prange) return Boolean;

    function Check_Nesting
       (tid: Task_ID; S: Positive) return Boolean;

  end Resource_Scheduler;
  protected body Resource_Scheduler is

    procedure Release (tid: Task_ID; S: Positive) is
    begin
      if(tid /= Locking_task(S)) then
        raise Release_Error;
      else
        Set_Priority(Preemption_Level(S),tid);
        Resource_Status(S) := False;
        DPCP_V(DPCP_T(S))  := DPCP_V(DPCP_T(S)) + 1;
        Locking_Task(S)    := Null_Task_ID;
	-- if current queue is empty, then switch queues
        if Queue_Ready then
          if Queue1'Count > 0 then
            Gate1 := TRUE;
          end if;  
        else 	
          if Queue2'Count > 0 then
            Gate2 := TRUE;
          end if;
        end if;              
      end if;
    end Release;

    entry Request
       (tid: Task_ID; S: Positive;  P: Positive; T: Time) when True is
           Task_Priority: Task_Prange := Get_Priority(tid);
    begin

      if (Task_Priority > Ceiling_Priority(S)) then 
		raise Ceiling_Error;
      elsif (Check_Nesting(tid, S) = False) then 
		raise Nesting_Error;
      elsif (Resource_Status(S) = not Locked) and then
            (Check_Ceiling(tid, Task_Priority)) and then
            (DPCP_V(DPCP_T(S)) > 1) then

          Resource_Status(S)  	:= Locked;
          Locking_Task(S) 	:= tid;
	  Task_P(S) 		:= P;
          Preemption_Level(S) 	:= Task_Priority;
          DPCP_V(DPCP_T(S))	:= DPCP_V(DPCP_T(S)) - 1;

          if(Resource_P(S) /= P) and then 
            (Task_Priority < Ceiling_PH(S)) then
            Active_Priority(S) := TASK_MAX_Priority;
            Set_Priority(TASK_MAX_Priority, tid); 
          elsif (Task_Priority < Ceiling_Priority(S)) then
              Active_Priority(S) := Ceiling_Priority(S);
              Set_Priority(Ceiling_Priority(S), Locking_task(S));      
	  else
            Active_Priority(S) := Task_Priority;
          end if;

      else
          if Queue_Ready then 	
            requeue Queue1;
          else 			
            requeue Queue2;
          end if;
      end if;
    end Request;

    function Check_Ceiling
       (tid: Task_ID; Task_Priority: Task_Prange) return Boolean is
    begin
      for I in 1 .. N loop
        if (Resource_Status(I) = Locked) and then
           (Locking_Task(I) /= tid) and then
             (Ceiling_Priority(I) >= Task_Priority) then  
		return False;
        end if;
      end loop;
      return True;
    end Check_Ceiling;

    function Check_Nesting
       (tid: Task_ID; S: Positive) return Boolean is
    begin
      for I in 1 .. N loop
        if  (Resource_Status(I) = True) and then
            (Locking_Task(I) = tid) then
		return False;
        end if;
      end loop;
      return True;
    end Check_Nesting;

    entry Queue1
      (tid: Task_ID; S: Positive; P: Positive; T:Time) when Gate1 is
        Task_Priority  : Task_Prange := Ada.Dynamic_Priorities.Get_Priority(tid);
    begin 
      ---------------------------------------------------------------------------
      if Queue1'Count = 0 then
        Gate1     := False;
        Queue_Ready := False;
      end if; 

      if (Resource_Status(S) = not Locked) and then 
            (Check_Ceiling(tid, Task_Priority)) and then
            (DPCP_V(DPCP_T(S)) > 1) then

          -- Resource is available, then lock it
          Resource_Status(S)  	:= Locked;
          Locking_Task(S) 	:= tid;
	  Task_P(S) 		:= P;
          Preemption_Level(S) 	:= Task_Priority;
          DPCP_V(DPCP_T(S))	:= DPCP_V(DPCP_T(S)) - 1;

          if(Resource_P(S) /= P) then 
            Active_Priority(S) := TASK_MAX_Priority;
            Set_Priority(TASK_MAX_Priority, tid);
	  else
            Active_Priority(S) := Task_Priority;
          end if;
      else
          -- resource is still not available, then requeue again
	  requeue Queue2;
      end if;
      ---------------------------------------------------------------------------
    end Queue1; 

    entry Queue2
      (tid: Task_ID; S: Positive; P: Positive; T:Time) when Gate2 is
        Task_Priority  : Task_Prange := Ada.Dynamic_Priorities.Get_Priority(tid);
    begin 
      ---------------------------------------------------------------------------
      if Queue2'Count = 0 then
        Gate2     := False;
        Queue_Ready := True;
      end if; 

      if (Resource_Status(S) = not Locked) and then 
            (Check_Ceiling(tid, Task_Priority)) and then
            (DPCP_V(DPCP_T(S)) > 1) then

          -- Resource is available, then lock it
          Resource_Status(S)  	:= Locked;
          Locking_Task(S) 	:= tid;
	  Task_P(S) 		:= P;
          Preemption_Level(S) 	:= Task_Priority;
          DPCP_V(DPCP_T(S))	:= DPCP_V(DPCP_T(S)) - 1;

          if(Resource_P(S) /= P) then 
            Active_Priority(S) := TASK_MAX_Priority;
            Set_Priority(TASK_MAX_Priority, tid);
	  else
            Active_Priority(S) := Task_Priority;
          end if;
      else
          -- resource is still not available, then requeue again
	  requeue Queue1;
      end if;
      ---------------------------------------------------------------------------
    end Queue2; 

  end Resource_Scheduler;

/*******************************************************************
 * CPU affinity (demo)
 ********************************************************************/

typedef int bool; 	
#define TRUE  1
#define FALSE 0 

void usage();                           	 /* Simple generic usage function */
bool do_cpu_stress(int numthreads);		 /* Entry point into CPU thrash   */
int  do_cpu_expensive_op(int myitem);            /* Single thread cpu intensive   */
bool check_cpu_expensive_op(int possible_result);/* Compare value to precomputed  */

int main( int argc, char **argv )
{
	int return_code = FALSE;

	/* Determine number of processors */
	int NUM_PROCS = sysconf(_SC_NPROCESSORS_CONF);
  	printf("System has %i processor(s).\n", NUM_PROCS);

	/* These need sane defaults, because the values will be used unless overriden */
	int num_cpus_to_spin = NUM_PROCS; 

        /* Check for user specified parameters */
	int option = 0; 
   	while ((option = getopt(argc, argv, "m:c:l?ahd")) != -1)
   	{
      		switch (option)
      		{
	       		case 'c': /* SPECIFY NUM CPUS TO MAKE BUSY */
		          num_cpus_to_spin = atoi(optarg);
			  if (num_cpus_to_spin < 1)
			  {
				printf("WARNING: Must utilize at least 1 cpu. Spinning "
					" all %i cpu(s) instead...\n", NUM_PROCS);
				num_cpus_to_spin = 1;
			  }
			  else if (num_cpus_to_spin > NUM_PROCS)
			  {
				printf("WARNING: %i cpu(s), are not "
				       "available on this system, spinning all %i cpu(s) "
                                       "instead...\n", num_cpus_to_spin, NUM_PROCS);
				num_cpus_to_spin = NUM_PROCS;
			  }
			  else
			  {	
			  	printf("Maxing computation on %i cpu(s)...\n",
			   	       num_cpus_to_spin);
			  }
			break;


			case '?':
	        		if (isprint (optopt))
	          		{
					fprintf (stderr, 
					"Unknown option `-%c'.\n", optopt);
				}
	        		else
				{
	          			fprintf (stderr,
	                   		"Unknown option character `\\x%x'.\n",
	                   		optopt);
				}
			break;

		        default:
		          usage(argv[0]);
		          exit(0);
		}
	}

	/* Kick off the actual work of spawning threads and computing */
	do_cpu_stress(num_cpus_to_spin); 
	return return_code;
}



/* This method simply prints the usage information for this program */
void usage()
{
	printf("[-c  NUM_CPUS_TO_STRESS]\n");
	printf("If no parameters are specified all cpu's will be made busy.\n");
	return;
}



/* This method creates the threads and sets the affinity. */
bool do_cpu_stress(int numthreads)
{
	int ret = TRUE;
	int created_thread = 0;

	/* We need a thread for each cpu we have... */
	while ( created_thread < numthreads - 1 )
	{
		int mypid = fork();
		
		if (mypid == 0) /* Child process */
 		{
		    printf("\tCreating Child Thread: #%i\n", created_thread);
		    break;
		}

		else /* Only parent executes this */
		{ 
		    /* Continue looping until we spawned enough threads! */ ;
		    created_thread++;
		} 
	}



	/* NOTE: All threads execute code from here down! */



	cpu_set_t mask;

	/* CPU_ZERO initializes all the bits in the mask to zero. */ 
        CPU_ZERO( &mask ); 	

	/* CPU_SET sets only the bit corresponding to cpu. */
        CPU_SET( created_thread, &mask );  
        
	/* sched_setaffinity returns 0 in success */
        if( sched_setaffinity( 0, sizeof(mask), &mask ) == -1 )
	{
		printf("WARNING: Could not set Affinity\n");
	}

	 int computation_res = do_cpu_expensive_op(41);
	 cpu_set_t mycpuid;	 
	 sched_getaffinity(0, sizeof(mycpuid), &mycpuid);

	 if ( check_cpu_expensive_op(computation_res) )
	 {
		printf("SUCCESS: Thread completed computational task, and PASSED integrity check!\n",
			mycpuid);
		ret = TRUE;	
	 }
	 else 
	 {
		printf("FAILURE: Thread failed integrity check!\n",
			mycpuid);
		ret = FALSE;
	 }	


	return ret;
} 


/* computationally wasteful fibonaci sequence finder 
   we do not store known computed values. 
*/
int do_cpu_expensive_op(int myitem)
{
	/* FIXME: Should check myitem size because this could overflow quick */
	if (myitem == 0 || myitem == 1) 
	{ 
		return myitem; 
	}

	return ( do_cpu_expensive_op( myitem - 1 ) + do_cpu_expensive_op( myitem - 2 ) ); 
}



/* This method simply takes an integer argument
   and compares it to a precomputed correct value.
*/
bool check_cpu_expensive_op(int possible_result)
{
	/* 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987 ... fib(41) = 165580141 */ 
	int actual_result = 165580141;
	return ( actual_result == possible_result );
}


/* 
 * Periodic threads
 */

struct periodic_data {
  struct timespec per;
  int sig_num;
};

/* Periodic thread that creates a periodic timer */
void * periodic (void *arg)
{
  struct periodic_data my_data;
  siginfo_t received_sig;
  struct itimerspec timerdata;
  timer_t timer_id;
  struct sigevent event;
  sigset_t set;
  int activation_count = 0;


  my_data = * (struct periodic_data*)arg;

  /* Create timer */
  event.sigev_notify = SIGEV_SIGNAL;
  event.sigev_signo = my_data.sig_num;
  CHKE_FATAL( timer_create (CLOCK_REALTIME, &event, &timer_id) );

  /* Arm periodic timer */
  timerdata.it_interval = my_data.per;
  timerdata.it_value = my_data.per;
  CHKE_FATAL( timer_settime (timer_id, 0, &timerdata, NULL) );

  sigemptyset (&set);
  sigaddset (&set, my_data.sig_num);
  /* The signal mask shall have been set by the parent */


  /* Wait timer and do "useful" work */
  while (1) {
    CHKE_FATAL( sigwaitinfo (&set, &received_sig) );

    printf ("Thread with period %ds%dns activation %d. Received sig:%d\n",
	    my_data.per.tv_sec, my_data.per.tv_nsec, ++activation_count,
	    received_sig.si_signo);
  }
}

/* Main program, that creates two periodic threads */
int main ()
{
     pthread_t t1,t2;
     pthread_attr_t attr;
     sigset_t set;
     struct periodic_data per_params1, per_params2;

     /* Set signal mask for main thread  */
     sigemptyset (&set);
     sigaddset (&set, SIGRTMIN);
     sigaddset (&set, SIGRTMIN+1);
     CHK( pthread_sigmask (SIG_BLOCK, &set, NULL) );

     CHK( pthread_attr_init (&attr) );

     /* first thread */
     per_params1.per.tv_sec = 0; per_params1.per.tv_nsec = 500000000;
     per_params1.sig_num = SIGRTMIN;
     CHK( pthread_create (&t1, &attr, periodic, &per_params1) );
     
     /* second thread */
     per_params2.per.tv_sec = 1; per_params2.per.tv_nsec = 500000000;
     per_params2.sig_num = SIGRTMIN+1;
     CHK( pthread_create (&t2, &attr, periodic, &per_params2) );
     
     /* Allows threads to execute for a while */
     sleep (20); 
     exit (0);
}

/* 
 * Thread accepting the SIGINT signal and terminates the process
 */

static int error_status=-1;

void * sigwaiter (void * arg)
{
  sigset_t set;
  siginfo_t siginfo;
  int counter = 0;

  sigemptyset (&set);
  sigaddset (&set, SIGINT);
  /* The signal mask is inherited from the parent */
  /* It must have SIGINT masked */

  while (1) {
    if (sigwaitinfo (&set, &siginfo)) {
      /*error*/
      perror ("sigwaitinfo");
      pthread_exit ((void*)&error_status);
    }
    if (siginfo.si_signo == SIGINT) {
      counter ++;
      if (counter < 4) {
	printf ("SIGINT %d received. signo: %d, si_code:%d, si_value:%d\n", 
		counter, siginfo.si_signo, siginfo.si_code, 
		siginfo.si_value.sival_int);
      } else {
	printf ("Final signal received\n");
	exit(0);
      }
    } else {
      pthread_exit ((void *)&error_status);
    }
  }
}


/* Main thread */

int main ()
{
  sigset_t set;
  pthread_t th;
  union sigval value = {0};
  struct sigaction action;
  int ret;

  sigemptyset(&set);
  sigaddset(&set, SIGINT);
  if ((ret = pthread_sigmask(SIG_BLOCK, &set, NULL)))
    printf ("pthread_sigmask:%s\n", strerror (ret));

  action.sa_handler = SIG_DFL;
  sigemptyset(&action.sa_mask);
  action.sa_flags = SA_SIGINFO;
  if (sigaction (SIGINT, &action, NULL))
    perror ("sigaction");

  /* Create task */
  if ((ret = pthread_create(&th, NULL, sigwaiter, NULL)))
    printf ("Error in thread creation:%s\n", strerror (ret));

  while (1) {
    value.sival_int++;
    printf ("Main sends signal %d\n", value.sival_int);
    if (sigqueue (0, SIGINT, value))
      perror ("sigqueue");
    sleep(10);
  }
}


// producer_consumer test case

struct buffer {
  int num_consumed;
  int num_produced;
  pthread_cond_t cond;
  pthread_mutex_t mutex;
  int data[20]; 
};

void * producer (void *arg)
{
  struct buffer *my_buffer;
  int i;

  my_buffer = (struct buffer *)arg;
  for (i=40; i>20; i--) {
    eat (0.3);
    printf ("producer produces i=%d\n", i);
    CHK( pthread_mutex_lock (&my_buffer->mutex) );
    my_buffer->num_produced++;
    my_buffer->data[my_buffer->num_produced-1] = i;
    CHK( pthread_cond_signal (&my_buffer->cond) );
    CHK( pthread_mutex_unlock (&my_buffer->mutex) );
  }
  pthread_exit (NULL);
  return NULL;
}

void * consumer (void *arg)
{
  struct buffer *my_buffer;
  int i,consumed;

  my_buffer = (struct buffer *)arg;
  for (i=0; i<20; i++) {
    eat (0.5);
    CHK( pthread_mutex_lock (&my_buffer->mutex) );
    while (my_buffer->num_produced <= my_buffer->num_consumed) {
      CHK( pthread_cond_wait (&my_buffer->cond, &my_buffer->mutex) );
    }
    my_buffer->num_consumed++;
    consumed = my_buffer->data[my_buffer->num_consumed - 1];
    CHK( pthread_mutex_unlock (&my_buffer->mutex) );
    printf ("consumer consumes i=%d\n", consumed);
  }
  pthread_exit (NULL);
  return NULL;
}

int main ()
{
  pthread_mutexattr_t mutexattr;
  pthread_condattr_t condattr;
  pthread_attr_t pthreadattr;
  pthread_t t1, t2;
  struct buffer my_buffer;

  adjust ();

  // Init buffer
  my_buffer.num_produced = 0;
  my_buffer.num_consumed = 0;
  CHK( pthread_mutexattr_init (&mutexattr) );
  CHK( pthread_mutex_init (&my_buffer.mutex, &mutexattr) );
  
  CHK( pthread_condattr_init (&condattr) );
  CHK( pthread_cond_init (&my_buffer.cond, &condattr) );
  

  // Create threads
  CHK( pthread_attr_init (&pthreadattr) );
  CHK( pthread_attr_setschedpolicy (&pthreadattr, SCHED_RR) );

  CHK( pthread_create (&t1, &pthreadattr, producer, &my_buffer) );
  CHK( pthread_create (&t2, &pthreadattr, consumer, &my_buffer) );
  
  CHK( pthread_join (t1, NULL) );
  CHK( pthread_join (t2, NULL) );

  return 0;
}