Injection Time

This signifies the time at which the faults are to be injected based on the configurations. You have the following options for fault injection time:

  • Current : Here, system captures the current time and injects havocs for the specified duration from current date and time.
  • Specified : Here, you need to specify the exact date and time and duration (in HH:MM:SS format) at which the configured fault is to be injected.
  • Random : You can specify the start date and time and end date and time along with the duration (in HH:MM:SS format) for injecting faults. The system picks any random time between the specified time and injects faults based on the provided duration.

Duration

This section displays information about the total duration of the injected havoc. This information is divided into three stages – Shoot Up, Stable, and Tumble. This information is also displayed in the form of a graph.

  • Shoot Up : This is the ramp-up phase.
  • Stable: This is the duration phase.
  • Tumble: This is the ramp-down phase.

Server Info

  • Tier : The tier on which havoc is to be injected.
  • Server Selection Mode : Choose if you want to select a specified or a dynamic server.
    • Specified : You can select specific server(s).
    • Dynamic : You can select one or more servers from the list.
      • ALL : If you select this option, all the servers are selected.
      • ANY : Provide any number of servers.
      • ANY% : Provide any percentage of available servers.
  • Connection Type : Havocs are injected on the server through these connection types (modes) – SSH and Cavisson Agent.
  • User Type : The type of user by which you want to inject the havoc (Default User, Switch User without sudo, or Switch User with sudo).
  • Havoc User : The username as per the user type selected earlier.

Kubernetes Info

  • Cluster : Firstly, configure the cluster from Monitor. After configuring the cluster, select the cluster containing the pods from the dropdown where your target containers are running. A cluster is a set of nodes that runs the containerized applications in Kubernetes.
  • Namespace : According to the cluster selected, Namespace is filtered, and you can select the Namespace to filter the visible set of pods to inject. \n Using the drop-down menus accordingly, select one or more of your Deployments, StatefulSets, ReplicaSets, DaemonSets, or Pods where you want to cause havoc.
  • Pod Lists : Preview the targeted pods on which havoc will be injected on the right side of Kubernetes Info.
    1. Specified : Select a specified Pod to inject havoc.
    2. ALL : Select all eligible Pods to inject havoc.
    3. ANY : Select a specified percentage of Pods from any eligible Pods available to inject havoc.
    4. PATTERN : Select a specified pattern-based Pods from any eligible Pods available to inject havoc.
    5. Container List : Select Container(s) which you want to target.

Containers Info

  • Tier : Select the tier name where the required server is running.
  • Server : Specify the CMON server (Cavisson Agent) where the required containers are running.
  • Containers name : Specify the containers where you want to inject havocs. You can select multiple containers running on the selected server.

CPU Burst

  • CPU Attack : This is the way you want to consume the CPU resources.
  • Fill Core : The number of cores in the CPU that you want to consume. The maximum limit to fill the cores is N-1, where N is the total number of cores available in the system.
  • CPU Utilization % : Burns CPU on mentioned server(s). The default value is 10% and maximum can be 100%.
  • CPU Cores (When 'Fill Core' is selected as CPU Attack) : Number of CPU cores to be consumed.
  • CPU Burn (When 'CPU Utilization %' is selected as CPU Attack) : CPU % to be burnt.

Disk Swindle

  • Partition Name : Partition name of the server.
  • Partition value : % of partition to be filled (between 1% and 100%). If you provide the partition value more than 90%, a confirmation message is displayed additionally.

Disk Swindle (Windows)

  • Drive Name : Drive name of the server.
  • Drive value : % of drive to be filled (between 1% and 100%). If you provide the partition value more than 90%, a confirmation message is displayed additionally.

I/O Shoot Up

  • Operation : The I/O operation to be performed. By selecting either 'Read' or 'Write' operations on the blocks based on user-defined Block Size.
  • Workers : Number of workers to run in parallel.
  • Block Size : Block size for the selected operation.
  • Block Count : Block count for the selected operation.

Memory Outlay

  • Memory Unit : The unit in which memory is to be consumed (such as MB, GB, or percentage).
  • Target Memory : The value of the memory to be consumed.

Abort Application

  • Select Process By : User can select the Process by Process Name or Process ID or PID File.
  • Process ID : The ID of the process, which has to be killed by the havoc.
  • Process Name : Name of the process that user need to kill, It will search the process by given name and pick the first one if multiple match found.
  • PID File : Name of the PID File in which PID ID would be written,It will be the first line of the file and User need to give the absolute path of the file.
    Currently it is supported in the linux.
  • Value : The process value as per the process category (by Process Name or Process ID or PID File).

Terminates Cloud Instance

  • Cloud Type : The cloud server on which the instance is to be terminated (Azure or AWS). When you select 'Azure' as the cloud type:

When you select 'Azure' as the cloud type:

  • Computer Name : Cloud Server name/IP.
  • Resource Group Name : Logical containers for a collection of resources that can be treated as one logical instance.
  • User Name : Username for accessing the server.
  • Password : Password for accessing the server.

When you select 'AWS' as the cloud type:

  • Region Name : Name of the region for AWS server.
  • Access ID : A credential for accessing the AWS server.
  • Secret Access Key : A credential for accessing the AWS server.
  • Instance ID : The ID of the instance for AWS server.

When you select 'GCP' as the cloud type:

  • Google Service Account : A service account is an account for an application instead of an individual end user. Enter Google Service Account and upload Service Account Key.
  • Project List : Select Project List from drop down.
  • Instance Name : Select Instance from drop down.

Note : If Google Service Account is already configured, users can directly select from the drop down.

Common fields for all types

  • Operation : The type of operation in which you want to impact the cloud instance, such as terminate or reboot.
  • Terminate : Selecting this mode to terminate the cloud instance.
  • Reboot : Selecting this mode to reboot the cloud instance.
  • Delay : The duration (delay) in minutes after which the selected operation is performed.

Kill Server

  • Machine Type : The type of operating system, such as Linux.
  • Operation : The type of operation in which you want to impact the machine, such as shut down or reboot.
  • Shut Down : Selecting this mode kills the machine.
  • Reboot : Selecting this mode reboots the machine.
  • Delay : The duration (delay) in minutes after which the selected operation is performed.

Teleport

  • Travel to : Select whether you want to change the system time in the past or in the future from the current time.
  • Future : You can change the system time to future with respect to the defined duration.
  • Past : You can change the system time to past with respect to the defined duration.
  • Duration : Duration from the current time (in HH:MM:SS format).

Distort JMS queue targets

Distort JMS Queue havoc is used for sending lots of messages to check the resiliency of JMS server or sending incorrect messages for checking application behaviour.

JMS Type : In this, the JMS type is selected from the drop-down where you want to inject the havoc. JMS types are mainly of three types:
  • IBMMQ
  • Kafka
  • TIBCO
Configuring IBMMQ : You need to provide the following details within the JMS Configuration for IBMMQ: All fields are mandatory.
  • Server Name/ IP : This is the IBM MQ Server Name/ IP.
  • Port : This is the port where the IBM MQ is listening.
  • Queue Manager : This manages the resources associated with a queue that it owns.
  • Channel : This is used to transfer messages between queues.
  • Queue : This is responsible to hold the message until the receiver is ready.

For details refer to the links- Introduction to IBM WebSphere MQ - IBM Documentation

Configuring Kafka : You need to provide the following details within the JMS Configuration for Kafka:
  • Server Name/ IP : This is the Kafka Server IP / Hostname. It is a mandatory field.
  • Port : This is the port where Kafka is listening. It is a mandatory field.
  • Topic : This is used as a message-oriented middleware that is responsible to hold and deliver messages. It is a mandatory field.
Additional Settings for Kafka : To secure the connection, use SSL settings. You can configure the following:
  • SASL (Simple Authentication Security Layer) Settings :
  • Other Settings : When you select the Enable SASL Authentication checkbox, the system displays the following fields:
    1. User : Enter the SASL username.
    2. Password : Enter the SASL password.
    3. Mechanism : The SASL supports various mechanisms for authentication, such as"PLAIN ", "non-plaintext ", etc. Select mechanism as per the requirement.
    4. Message Key : The Message Key is a message identifier that helps to distinguish the clients as each client has a unique key.
    5. Header Key : Fill the header key in this field.
    6. Header Value : Fill the header value in this field.
    7. You have to fill the following fields in the SSL Settings:
    8. Use Specific Ciphers : From the dropdown, select the Cipher needed.
    9. Key File : Key files are open SSL generated keys with the crypto toolkit and saved into files with the. key or .pem extension. There is also a password for .key file.
    10. Trusted CA file : These are the trusted certificates, which are data files, used to cryptographically link an entity with a public key.

Once you click the Kafka is enforcing Client Authentication, you get another option/ field Certificate File in SSL Settings which are small data files that digitally bind a cryptographic key to an organization's detail.

For more details refer to the link- Apache Kafka

Configuring TIBCO : You need to provide the following details within the JMS Configuration for TIBCO:
  • Server Name IP : This is the TIBCO server IP/hostname. It is a mandatory field.
  • Port : This is the port where TIBCO is listening. It is a mandatory field.
  • Topic/Queue : Queue is responsible to hold the message until the receiver is ready. The topic is used as a message-oriented middleware that is responsible to hold and deliver messages. The user can either select Topic or Queue.
Additional Settings for TIBCO : In Additional Settings, you get the Other Settings option, After clicking the Other Security, you get the following fields:
  • Message Key : The Message Key is a message identifier that helps to distinguish the clients as each client has a unique key.
  • Header Value : Fill the header value in this field. Using SSL Settings You have to fill the following fields in the SSL Settings:
  • Using SSL Settings : You have to fill the following fields in the SSL Settings:
  • Use Specific Ciphers : From the dropdown, select the Cipher needed.
  • Trusted CA file : These are the trusted certificates, which are data files, used to cryptographically link an entity with a public key.

Once you click the Kafka is enforcing Client Authentication, you get another options/ fields Certificate File and Key File in SSL Settings

  • Certificate File and Key File in SSL Settings : Certificate File Field- which are small data files that digitally bind a cryptographic key to an organization's detail.
  • Key File : Key files are open SSL generated keys with the crypto toolkit and saved into files with the. key or .pem extension. There is also a password for .key file.

For more details refer to the link- JMS Message Models (tibco.com)

Intrude Network

  • Network Interface : It is the point of interconnection between a computer and a private or public network. The drop-down suggests all the interfaces that has been added on the server.
  • Corruption Percentage : The % of network packets needs to be corrupted or manipulated (the range is 10-50%).

Trim Network Packets

  • Network Interface : It is the point of interconnection between a computer and a private or public network. The drop-down suggests all the interfaces that has been added on the server.
  • Loss : The % of packets lost in the network (the range is 10-50%).

Dormant Network

  • Network Interface : It is the point of interconnection between a computer and a private or public network. The drop-down suggests all the interfaces that has been added on the server.
  • Mode : Select the mode whether you want to delay the network traffic by a specified amount of time or between a range of time.
  • Specified : Provide integer value of the field. The packets will be delayed by the specified amount of time.
  • Random : The packets will be delayed by any time between the two given values.

When the mode is 'Specified' :

  • Delay Value : The specified amount of time (in milliseconds) by which network traffic is to be delayed.

When the mode is 'Random' :

  • First Delay Value : The first value of the range of time (in milliseconds) by which network traffic is to be delayed.
  • Second Delay Value : The second (end) value of the range of time (in milliseconds) by which network traffic is to be delayed.

DNS Breakdown

  • Network Interface : It is the point of interconnection between a computer and a private or public network. The drop-down suggests all the interfaces that has been added on the server.
  • Server Mode : Select the mode for which you want to reject the calls on DNS server (such as Single, Range, or IP with CIDR).
  • IP/Host Address : The IP address for which you want to impact the network traffic.
  • CIDR Value (only when 'IP with CIDR' is the Server Mode) : The Classless Inter-Domain Routing (CIDR) value of the IP address.
  • Protocol : The protocol that you want to be impacted, such as TCP or UDP.

Interstellar

  • Network Interface : It is the point of interconnection between a computer and a private or public network. The drop-down suggests all the interfaces that has been added on the server.
  • Server Mode : Select the mode for which you want to reject the calls on DNS server (such as Single, Range, or IP with CIDR).
  • IP/Host Address : The IP address for which you want to impact the network traffic.
  • CIDR Value (only when 'IP with CIDR' is the Server Mode) : The Classless Inter-Domain Routing (CIDR) value of the IP address.
  • Protocol : The protocol that you want to be impacted, such as TCP, UDP, or ICMP.
  • Program Name : The program name is used to apply the user-provided rule on the specific program.
  • Traffic : Select the network traffic type you want to be impacted, such as ingress or egress.
  • Port : Port of the IP for which you want to impact the network traffic. Provide comma-separated multiple ports.

Alter Inbound Services

Induce delays in response time or service failures with exception on inbound services.

  • BT Name : From the drop down select the name of the business transaction of target request.
  • URL : Enter the relative URL of the target request.
  • Apply On : Specify the desired percentage of requests to be impacted. Default percentage is 100%.
  • Operation : Select the type of operation in which you want to Alter the Inbound Services either Delay Response Time or Service Failure.
  • Delay (When Delay Response Time is selected as Operation) : Enter the specified amount of time (in milliseconds) by which service failure is to be delayed.

Alter Outbound Services

Induce delays in response time or service failures with exception on inbound services.

  • BT Name : From the drop down select the name of the business transaction of target request.
  • URL : Enter the relative URL of the target request.
  • Protocol : Select protocol from drop down HTTP/non-HTTP. By Default HTTP is selected.
  • Integration Point (When HTTP is selected as Protocol) : Enter the name of integration point.
  • Host : Enter the Host name for integration point call out.
  • Apply On : Specify the desired percentage of requests to be impacted. Default percentage is 100%.
  • Operation : Select the type of operation in which you want to Alter the Outbound Services, such as Delay Response Time or Service Failure.
  • Delay (When Delay Response Time is selected as Operation) : Enter the specified amount of time (in milliseconds) by which service failure is to be delayed.

Method Invocation

Method Invocation Fault helps you simulate a condition where calls to a specific Java method can be delayed by a specific time or Method Failure with Exception.

  • BT Name : From the drop down select the name of the business transaction of target request.
  • URL : Enter the relative URL of the target request.
  • Method Name : Enter the fully qualified name of the java method.
  • Impact : Specify the desired percentage of requests to be impacted. Default percentage is 100%.

Heap Memory Leak

Memory leakage Havoc increases the Heap Utilization of JVM.

  • Leaks In : From the drop down select the unit in which memory is to be leaked (such as MB).
  • Leak Memory : Enter the value of the heap memory to be leaked.

Method Exception

  • BT Name : From the drop down select the name of the business transaction of target request.
  • URL : Enter the relative URL of the target request.
  • Method Name : Enter the fully qualified name of the java method.
  • Impact : Specify the desired percentage of requests to be impacted. Default percentage is 100%.

Havoc Delay

Delay is the time gap between the execution of two havocs. Delay will be not applicable to first havoc. It will be applicable from second havoc onwards.

Nethavoc supports two modes of delay(s):

  1. Delay with respect to the start of the previous havoc.
    For example, if havoc starts at 10:00 for duration of 5m and delay value in 2m, next havoc will be triggered at 10:02
  2. Delay with respect to the end of previous havoc.
    For example, if havoc starts at 10:00 for duration of 5m and delay value in 2m, next havoc will be triggered at 10:07

Duration & Iteration

Duration:

This section displays the total duration of the injected havoc. The maximum duration is 99hr:59m:59s.

Number of iterations:

It is the number of iterations to be performed over a given message set that can have one or more messages.

Delay In Scenario

Delay is the duration between the execution of two consecutive Havocs.

Application Kill

  • Kill Application: By injecting this havoc, application will be terminated. This fault can be injected based on the following options:
  • Directly: Select this option to kill the application immediately.
  • On method entry: Terminate application once a specific method is called and on its entry.
  • On Method Exit: Terminate the application once a specific method's execution is completed and on its exit.

Method Selection

  • Specific Method: Select the specific method from the drop down. Users can select multiple methods at a time.
  • Method Name: Enter the fully qualified name of the java method.

Exit Code

  • Exit codes categorize how an application termination process occurred, and it has four types of exit code: Normal, Abnormal, with Core Dump, and Custom processes post-termination, as explained below.
    • Normal: The application terminated successfully, indicating that it completed its tasks without encountering any significant issues.
    • Abnormal: This exit code signifies that the application terminated unexpectedly due to an error or exception. It could indicate issues such as invalid input, unexpected runtime conditions, or bugs in the application code.
    • With Core Dump: If the application terminates with a core dump, it suggests that a critical error occurred, leading to a crash. The core dump provides a snapshot of the application's memory at the time of the crash, which can be useful for debugging and identifying the cause of the issue.
    • Custom: Users can also give the custom exit code for application kill havoc. User will give the numerical value as custom exit code that is returned by a command or program to indicate its execution result. The exit codes range from 0 to 255, with 0 representing success, and 1-255 indicating various failure conditions.
  • Delay Before Kill: Add a delay before terminating or killing the application after the defined event or condition.

Application CPU Burst

  • CPU Attack : This is the way you want to consume the CPU resources.
    • Fill Millicore : The number of millicores in the CPU that you want to consume. The maximum limit to fill the millicores is N-1, where N is the total number of millicores available in the system.
      • CPU Millicores: Number of CPU millicores to be consumed.
  • CPU Utilization % : Burns CPU on mentioned server(s). The default value is 10% and maximum can be 100%.
  • CPU Burn : CPU % to be burnt.

Thread Leak

Simulate thread leaks on your application instance via this havoc. Following information is required:

  • Number of Thread: Enter the value to specify the number of generic threads that will be initiated in the application. The threads are executed simultaneously without impacting each other.
  • Sleep Time for each thread : Specify the sleep time (in ms) for each thread. Higher sleep times will use less CPU and higher sleep times have minimal effect on CPU, while lower sleep times increase CPU utilization.