Introduction
============

Accelerated IO SW library (XLIO) boosts the performance of applications written over standard socket API such as
web serving, reverse proxying, caching, load balancing, media streaming, and more. Reduction of latency, increasing
throughput and effective CPU utilization is achieved by full network stack bypass and direct access to
accelerated network hardware.
XLIO dynamically links with these applications at run-time, redirect standard socket API calls allowing them to be
be accelerated without modification.

XLIO execution modes
====================
XLIO supports two execution modes.
1 - Run to completion (R2C)
2 - Worker Threads

Run to completion
-----------------
By default, XLIO works in R2C mode, meaning, the execution context is
provided to XLIO by the application. In particular, XLIO code such as reading/writing packets,
polling CQs, handling sockets, etc, is performed as part of POSIX socket calls or Ultra API calls,
on the same thread which called the API.
In terms of performance this mode is the preferred one, however,
this mode requires from the application to work with sockets in an efficient way,
by avoiding sharing sockets between threads, provide enough execution time to XLIO, use per thread epoll and more.
In case of listen sockets, each thread should have its own listen socket.

Worker Threads
--------------
In this mode XLIO spawns XLIO worker threads. These threads run in the background and perform network operations.
This mode requires minimal network awareness from the aplication,
i.e applications may share sockets between threads, use single listening thread,
use single epoll context or rarely call socket APIs.
While R2C mode provides best performance, Worker Threads mode provides greater flexibility.
The number of XLIO worker threads is controlled by the performance.threading.worker_threads parameter.
XLIO Ultra API is not supported with this mode.
Please see User Manual for additional details and current limitations.

Configuration Subsystem
=======================

On default startup the XLIO library logs to stderr the version, the modified
configuration parameters being used and their values.
Please notice that except monitor.log.level, library logs just those parameters whose value != default.

Example:
 XLIO INFO   : ---------------------------------------------------------------------------
 XLIO INFO   : XLIO_VERSION: 1.0.0-0 Development Snapshot built on May 26 2021 17:00:30
 XLIO INFO   : Git: 46d203af1d322799c8de5789ba4fe0955f8d9942
 XLIO INFO   : Cmd Line: uname -r
 XLIO INFO   : Current Time: Wed May 26 17:02:52 2021
 XLIO INFO   : Pid: 31535
 XLIO INFO   : OFED Version: MLNX_OFED_LINUX-5.2-0.4.8.0:
 XLIO DEBUG  : System: 4.18.0-80.el8.x86_64
 XLIO INFO   : Architecture: x86_64
 XLIO INFO   : Node: r-aa-zorro006
 XLIO INFO   : ---------------------------------------------------------------------------

Configuration Values
====================

XLIO exposes runtime configuration parameters organized into hierarchical groups.

For the complete reference -- including types, defaults, environment variables, and
constraints -- see docs/xlio_config_reference.md

XLIO Monitoring & Performance Counters
=====================================
The XLIO internal performance counters include information per user
sockets and a global view on select() and epoll_wait() usage by the application.

Use the 'xlio_stats' included utility to view the per socket information and
performance counters during run time.
Usage:
        xlio_stats [-p pid] [-k directory] [-v view] [-d details] [-i interval]

Defaults:
        find_pid=enabled, directory="/tmp/", view=1, details=1, interval=1,

Options:
  -p, --pid=<pid>               Show XLIO statistics for process with pid: <pid>
  -k, --directory=<directory>   Set shared memory directory path to <directory>
  -n, --name=<application>      Show XLIO statistics for application: <application>
  -f, --find_pid                Find and show statistics for XLIO instance running (default)
  -F, --forbid_clean            By setting this flag inactive shared objects would not be removed
  -i, --interval=<n>            Print report every <n> seconds
  -c, --cycles=<n>              Do <n> report print cycles and exit, use 0 value for infinite (default)
  -v, --view=<1|2|3|4|5|6>      Set view type:
                                1 - Basic info
                                2 - Extra info
                                3 - Full info
                                4 - Multicast groups
                                5 - Show as 'netstat -tunaep'
                                6 - Entity Context info
  -d, --details=<1|2>           Set details mode:
                                1 - Totals
                                2 - Deltas
  -z, --zero                    Zero counters
  -l, --log_level=<level>       Set XLIO log level to <level>(1 <= level <= 7)
  -S, --fd_dump=<fd> [<level>]  Dump statistics for fd number <fd> using log level <level>. use 0 value for all open fds
  -D, --details_level=<level>   Set XLIO log details level to <level>(0 <= level <= 3)
  -s, --sockets=<list|range>    Log only sockets that match <list> or <range>, format: 4-16 or 1,9 (or combination)
  -V, --version                 Print version
  -h, --help                    Print this help message


Use monitor.stats.file_path to get internal XLIO statistics like xlio_stats provide.
If this parameter is set and the user application performed transmit or receive
activity on a socket, then these values will be logs once the sockets are closed.

Below is a logout example of a socket performance counters.
Below the logout example there is some explanations about the numbers.

XLIO: [fd=10] Tx Offload: 455 KB / 233020 / 0 / 3 [bytes/packets/drops/errors]
XLIO: [fd=10] Tx OS info:   0 KB /      0 / 0 [bytes/packets/errors]
XLIO: [fd=10] Rx Offload: 455 KB / 233020 / 0 / 0 [bytes/packets/eagains/errors]
XLIO: [fd=10] Rx byte: max 200 / dropped 0 (0.00%) / limit 2000000
XLIO: [fd=10] Rx pkt : max 1 / dropped 0 (0.00%)
XLIO: [fd=10] Rx OS info:   0 KB /      0 / 0 [bytes/packets/errors]
XLIO: [fd=10] Rx poll: 0 / 233020 (100.00%) [miss/hit]

Looking good :)
- No errors on transmit or receive on this socket (user fd=10)
- All the traffic was offloaded. No packets transmitted or receive via the OS.
- Just about no missed Rx polls (see performance.polling.blocking_rx_poll_usec & 
 performance.polling.iomux.poll_usec), meaning
 the receiving thread did not get to a blocked state to cause a contexts
 switch and hurt latency.
- No dropped packets caused by socket receive buffer limit (see XLIO_RX_BYTES_MIN).
- No 'No buffers' errors in Buffer Pools.
- No 'HW RX Packets' drops in CQs.

Interrupt Moderation
====================
The basic idea behind interrupt moderation is that the HW will not generate
interrupt for each packet, but instead only after some amount of packets received
or after the packet was held for some time.

The adaptive interrupt moderation change this packet count and time period
automatically to reach a desired rate of interrupts.


1. Use performance.polling.blocking_rx_poll_usec=0 and 
    performance.polling.iomux.poll_usec=0 to work in interrupt driven mode.

2. Control the period and frame count parameters with:
    performance.completion_queue.interrupt_moderation.packet_count - hold #count frames before interrupt
    performance.completion_queue.interrupt_moderation.period_usec - hold #usec before interrupt

3. Control the adaptive algorithm with the following:
    performance.completion_queue.interrupt_moderation.adaptive_count - max possible #count frames to hold
    performance.completion_queue.interrupt_moderation.adaptive_period_usec - max possible #usec to hold
    performance.completion_queue.interrupt_moderation.adaptive_interrupt_per_sec - desired interrupt rate
    performance.completion_queue.interrupt_moderation.adaptive_change_frequency_msec - frequency of adaptation

4. Disable CQ moderation with performance.completion_queue.interrupt_moderation.enable=false
5. Disable Adaptive CQ moderation with 
   performance.completion_queue.interrupt_moderation.adaptive_change_frequency_msec=0

Install library from rpm or debian
=================================

Installing:
Install the package as any other rpm or debian package [rpm -i libxlio.X.Y.Z-R.rpm].
The installation copies the XLIO library to: /usr/lib[64]/libxlio.so
The XLIO monitoring utility is installed at: /usr/bin/xlio_stats
The XLIO extra socket API is located at: /usr/include/mellanox/xlio_extra.h

Upgrading:
Use rpm update procedure: # rpm -U libxlio.X.Y.Z-R.rpm
You can upgrade by uninstalling (rpm -e) the previously installed package
before starting to install the new library rpm.

Uninstalling:
When uninstalling remember to uninstall (rpm -e) the package before you uninstall DOCA.

Troubleshooting
===============

* High log level:

 XLIO WARNING: *************************************************************
 XLIO WARNING: * XLIO is currently configured with high log level          *
 XLIO WARNING: * Application performance will decrease in this log level!  *
 XLIO WARNING: * This log level is recommended for debugging purposes only *
 XLIO WARNING: *************************************************************

This warning message means that you are using XLIO with high log level:
monitor.log.level variable value is set to 4 or more.
In order to fix it - set monitor.log.level to it's default value: 3

* CAP_NET_RAW and root access

 XLIO_WARNING: ******************************************************************************
 XLIO_WARNING: * Interface <Interface Name> will not be offloaded.
 XLIO_WARNING: * Offloaded resources are restricted to root or user with CAP_NET_RAW privileges
 XLIO_WARNING: * Read the CAP_NET_RAW and root access section in the XLIO's User Manual for more information
 XLIO_WARNING: ******************************************************************************

This warning message means that XLIO tried to create a hardware QP resource
while the kernel requires this operation to be performed only by privileged
users. Run as user root or grant CAP_NET_RAW privileges to your user

* Huge pages out of resource:

 XLIO WARNING: ************************************************************
 XLIO WARNING: NO IMMEDIATE ACTION NEEDED!
 XLIO WARNING: Not enough suitable hugepages to allocate 2097152 kB.
 XLIO WARNING: Allocation will be done with regular pages.
 XLIO WARNING: To avoid this message, either increase number of hugepages
 XLIO WARNING: or switch to a different memory allocation type:
 XLIO WARNING:   core.resources.hugepages.enable=false
 XLIO INFO   : Hugepages info:
 XLIO INFO   :   1048576 kB : total=0 free=0
 XLIO INFO   :   2048 kB : total=0 free=0
 XLIO WARNING: ************************************************************

This warning message means that you are using XLIO with hugepages memory allocation,
but not enough huge pages resources are available in the system.
If you want XLIO to take full advantage of the performance benefits of huge pages then
you should restart the application after adding more hugepages resources in your
system or trying to free unused hupepages shared memory segments with the below script.

NOTE: Use 'ipcs -m' and 'ipcrm -m shmid' to check and clean unused shared memory segments.
Below is a short script to help you release XLIO unused huge pages resources:
    for shmid in `ipcs -m | grep 0x00000000 | awk '{print $2}'`;
    do echo 'Clearing' $shmid; ipcrm -m $shmid;
    done;

For more information, refer to the "HugeTLB Pages" documentation of the Linux kernel.

* Not supported Bonding Configuration:

 XLIO WARNING: ******************************************************************************
 XLIO WARNING: XLIO doesn't support current bonding configuration of bond0.
 XLIO WARNING: The only supported bonding mode is "802.3ad(#4)" or "active-backup(#1)"
 XLIO WARNING: with "fail_over_mac=1" or "fail_over_mac=0".
 XLIO WARNING: The effect of working in unsupported bonding mode is undefined.
 XLIO WARNING: Read more about Bonding in the XLIO's User Manual
 XLIO WARNING: ******************************************************************************

This warning message means that XLIO has detected bonding device which is configured
to work in mode which is not supported by XLIO, this means that XLIO will not support
high availability events for that interface.
XLIO currently supports just active-backup(#1) or 802.3ad(#4) and fail_over_mac = 1 or 0 mode.
In order to fix this issue please change the bonding configuration.

Example:

Lets assume that the bonding device is bond0, which has two slaves: ib0 and
ib1.

Shut down the bond0 interface:
#ifconfig bond0 down

Find all the slaves of bond0:
#cat sys/class/net/bond0/bonding/slaves
ib0 ib1

Free all the slaves:
#echo -ib0 > /sys/class/net/bond0/bonding/slaves
#echo -ib1 > /sys/class/net/bond0/bonding/slaves

Change the bond mode:
#echo active-backup > /sys/class/net/bond0/bonding/mode

Change the fail_over_mac mode:
#echo 1 > /sys/class/net/bond0/bonding/fail_over_mac

Enslave the interfaces back:
#echo +ib0 > /sys/class/net/bond0/bonding/slaves
#echo +ib1 > /sys/class/net/bond0/bonding/slaves

Bring up the bonding interface:
#ifconfig bond0 up
OR
#ifconfig bond0 <ip> netmask <netmask> up

* Not supported Bonding & VLAN Configuration:

 XLIO WARNING: ******************************************************************
 XLIO WARNING: bond0.10: vlan over bond while fail_over_mac=1 is not offloaded
 XLIO WARNING: ******************************************************************

This warning message means that XLIO has detected bonding device which is configured with
VLAN over it while fail_over_mac=1.
This means that the bond will not be offloaded.
In order to fix this issue please change the bonding configuration.
