Configuration Management

This piece of the FI framework allows you to set expectations for faults to happen. The settings can be applied either statically (in advance) or in runtime. The desired level of faults in the framework can be configured two ways:

• editing src/aop/fi-site.xml configuration file. This file is similar to other Hadoop's config files
• setting system properties of JVM through VM startup parameters or in build.properties file

Probability Model

This is fundamentally a coin flipper. The methods of this class are getting a random number between 0.0 and 1.0 and then checking if a new number has happened in the range of 0.0 and a configured level for the fault in question. If that condition is true then the fault will occur.

Thus, to guarantee the happening of a fault one needs to set an appropriate level to 1.0. To completely prevent a fault from happening its probability level has to be set to 0.0.

Note: The default probability level is set to 0 (zero) unless the level is changed explicitly through the configuration file or in the runtime. The name of the default level's configuration parameter is fi.*

Fault Injection Mechanism: AOP and AspectJ

The foundation of Hadoop's FI framework includes a cross-cutting concept implemented by AspectJ. The following basic terms are important to remember:

• A cross-cutting concept (aspect) is behavior, and often data, that is used across the scope of a piece of software
• In AOP, the aspects provide a mechanism by which a cross-cutting concern can be specified in a modular way
• Advice is the code that is executed when an aspect is invoked
• Join point (or pointcut) is a specific point within the application that may or not invoke some advice

Existing Join Points

The following readily available join points are provided by AspectJ:

• Join when a method is called
• Join during a method's execution
• Join when a constructor is invoked
• Join during a constructor's execution
• Join during aspect advice execution
• Join before an object is initialized
• Join during object initialization
• Join during static initializer execution
• Join when a class's field is referenced
• Join when a class's field is assigned
• Join when a handler is executed

How to Use the Fault Injection Framework

Faults can be triggered as follows:

• During runtime:

      % ant run-test-hdfs -Dfi.hdfs.datanode.BlockReceiver=0.12

  To set a certain level, for example 25%, of all injected faults use:

      % ant run-test-hdfs-fault-inject -Dfi.*=0.25

• From a program:

      package org.apache.hadoop.fs;
  
      import org.junit.Test;
      import org.junit.Before;
  
      public class DemoFiTest {
        public static final String BLOCK_RECEIVER_FAULT="hdfs.datanode.BlockReceiver";
        @Override
        @Before
        public void setUp() {
          //Setting up the test's environment as required
        }
  
        @Test
        public void testFI() {
          // It triggers the fault, assuming that there's one called 'hdfs.datanode.BlockReceiver'
          System.setProperty("fi." + BLOCK_RECEIVER_FAULT, "0.12");
          //
          // The main logic of your tests goes here
          //
          // Now set the level back to 0 (zero) to prevent this fault from happening again
          System.setProperty("fi." + BLOCK_RECEIVER_FAULT, "0.0");
          // or delete its trigger completely
          System.getProperties().remove("fi." + BLOCK_RECEIVER_FAULT);
        }
  
        @Override
        @After
        public void tearDown() {
          //Cleaning up test test environment
        }
      }

As you can see above these two methods do the same thing. They are setting the probability level of hdfs.datanode.BlockReceiver at 12%. The difference, however, is that the program provides more flexibility and allows you to turn a fault off when a test no longer needs it.