This README is for OpenSM and the diagnostic utilities
in this directory (userspace/management).

Tree structure

userspace/management
	README

	doc/
	libibcommon/
		include/infiniband/
		src/
	libibmad/
		include/infiniband/
		src/
	libibumad/
		include/infiniband/
		src/
	diags/
		ibstat/
			src/
		ibaddr/
			src/
		ibnetdiscover/
			src/
		ibping/
			src/
		ibroute/
			src/
		ibsysstat/
			src/
		ibtracert/
			src/
		perfquery/
			src/
		sminfo/
			src/
		smpdump/
			src/
		smpquery/
			src/
		scripts/
	osm/
		complib/
		doc/
		include/
			complib/
			iba/
			opensm/
			vendor/
		libvendor/
		opensm/

Building

To make this tree, 
1. In libibcommon, libibumad, and libibmad (in that order), run:
./autogen.sh && ./configure && make && make install
2. In all diags, and osm subdirectories, run:
./autogen.sh && ./configure
3. At top level of management, run:
make && make install 

Typically the autogen and configure steps only need be done the first
time unless configure.in or Makefile.am changes in the directories.

Libraries are installed by default at /usr/local/lib and binaries at 
/usr/local/bin. 

Note that autotools is implemented for the libraries at the management 
level, the diags subdirectories, and the OpenSM and its
libraries (libosmcomp and libosmvendor).


Running

After compiling and installing, you can run opensm by invoking
/usr/local/bin/opensm &

opensm must be run as root.

Note also that you must have udev mount /dev/infiniband or do it manually.
See .../src/linux-kernel/docs/user_mad.txt. Also, ib_umad module must be
loaded.

opensm will run on the first existing port on the first IB device (HCA).
You can override that by using "-g <portguid_in_hex>". 
Verify that the first port is active. This assumes the port is plugged into
another IB device.

In case of problems, run the opensm with -V and send the log file
(/var/log/osm.log).


IMPORTANT:
Don't forget to modprobe ib_umad and make sure udev is configured before using 
any of the userspace programs.


Current OpenIB OpenSM Limitations:
1. Multipacket RMPP is not supported (so SA GetTable truncates
the response to fit into a single MAD) with ABI version 2 but
is supported with ABI version 3 and beyond of user_mad module
(and libibumad library).
2. Retry mechanism in SM is primitive and needs enhancing to deal with 
ports which are active but don't respond to SM MADs.
3. Async events are not yet supported (by OpenSM). The only one supported is
local LID change (and this is handled in the mthca driver). Future versions
of OpenSM will need to act on more local events.


Tuning OpenSM for Large Clusters 

Currently OpenSM is compiled with debug and no optimization. This
should be changed to at least -O2 (and perhaps -O4) but I would start
with -O2. This results in a 2x speedup for some code paths.

OpenSM supports a pipelining mode for SMPs. The default is 1
outstanding SMP. -maxsmps <#> indicates the number of outstanding SMPs
allowed and should speed up the initialization. Useful values of this
are 16 and 32.

Beyond this, there may be some issue with a link which is causing
timeout and retries to kick in. The OSM log should have some messages in
there indicating this.

Other utilites:
ibstat - show host adapters status
ibstatus - similar to ibstat but implemented as a script
ibnetdiscover - scan topology
ibaddr - shows the lid range and default GID of the target (default is
	the local port)
ibroute - display unicast and multicast forwarding tables of switches
ibtracert - display unicast or multicast route from source to destination
ibping - ping/pong between IB nodes (currently using vendor MADs)
ibsysstat - obtain basic information on remote node
	(hostname, cpus, memory, utilization) 
sminfo - query the SMInfo attribute on a node
smpdump - simple solicited SMP query tool. Output is hex dump
	(unless requested otherwise, e.g. using -s)
smpquery - formatted SMP query tool
perfquery - dump (and optionally clear) the performance (including error)
	counters of the destination port
ibcheckport - perform some basic tests on the specified port
ibchecknode - perform some basic tests on the specified node
ibcheckerrors - check if the error counters of the port/node have passed
	some predefined thresholds
ibchecknet - perform port/node/errors check on the subnet. ibnetdiscover
	output can be used as in input topology
ibswitches - scan the net or use existing net topology file and list all
	switches
ibhosts - scan the net or use existing net topology file and list all hosts
