*** Copyright Notice ***
 
BeStMan Copyright (c) 2010-2012, The Regents of the University of California, th
rough Lawrence Berkeley National Laboratory (subject to receipt of any required 
approvals from the U.S. Dept. of Energy).  This software was developed under fun
ding from the U.S. Department of Energy and is associated with the Berkeley Lab 
Scientific Data Management Group projects. All rights reserved.

If you have questions about your rights to use or distribute this software, plea
se contact Berkeley Lab's Technology Transfer Department at  TTD@lbl.gov.

NOTICE.  This software was developed under funding from the U.S. Department of E
nergy.  As such, the U.S. Government has been granted for itself and others acti
ng on its behalf a paid-up, nonexclusive, irrevocable, worldwide license in the 
Software to reproduce, prepare derivative works, and perform publicly and displa
y publicly.  Beginning five (5) years after the date permission to assert copyri
ght is obtained from the U.S. Department of Energy, and subject to any subsequen
t five (5) year renewals, the U.S. Government is granted for itself and others a
cting on its behalf a paid-up, nonexclusive, irrevocable, worldwide license in t
he Software to reproduce, prepare derivative works, distribute copies to the pub
lic, perform publicly and display publicly, and to permit others to do so.

#################################################

Berkeley Storage Manager (BeStMan) v2.2.2.x (a.k.a. BeStMan2 v2.2.x)
BeStMan SRM Clients
SRM-Tester
http://sdm.lbl.gov/bestman
Questions: srm@lbl.gov

BeStMan LICENSE: https://sdm.lbl.gov/wiki/Software/BeStMan/License
BeStMan COPYRIGHT: https://sdm.lbl.gov/wiki/Software/BeStMan/Copyright

Note: Support is not available for the sources.

Dependency requirements for BeStMan setup are below, and 
each software package follows its own license and copyright:

JDK 1.6.0_27 or 1.7.0
Apache ant 1.8.2
Axis 1.4
WSDL4j-1.6.2
Xerces-2.11.0
Jetty 8.0.1.v20110908
Java CoG JGlobus 1.8.0
Java Servlet API 3.0
Slf4j-1.6.2
Berkeley DB 4.1.10
which4j-0.21
vomsjapi 2.0.6
commons collections 3.2.1
commons lang 2.6
esapi 2.0.1
joda-time 1.6.2
opensaml 2.5.2
openws 1.4.2 3
privilege-xacml 2.2.5
velocity 1.7
Xalan 2.7.1
XML-security 1.4.5
Bouncy Castle crypto lib 1.4.6

#####################################################################
**** To make a setup deployment
############################################################################

1. Make sure you have the same build.properties from the sources build
2. "ant" will deploy executable scripts and configuration files to the install.root 
3. Adjustment to the configuration values can be done by the following two options
   a) manually editing dist/etc/bestman2
   b) using the configure options described in the following section.
   

#####################################################################
**** To make a custom configuration
############################################################################

1. Make sure you have Oracle/SUN or IBM Java 1.6.0_01 or higher installed
   OpenJDK 1.7.0 would be ok too.

2. ./configure --help  # to see the options
   Sample configure command options:

  e.g.1. % ./configure \
            --enable-gateway-mode \
            --with-globus-tcp-port-range=6201,6299 \
            --with-https-port=6288 \
            --with-eventlog-path=/data/bestman/log \
            --with-cachelog-path=/data/bestman/log \
            --with-certfile-path=/etc/grid-security/hostcert.pem \
            --with-keyfile-path=/etc/grid-security/hostkey.pem \
            --enable-sudofsmng \
            --enable-gums \
            --with-gums-url="https://gumsserver.lbl.gov:8443/gums/services/GUMSXACMLAuthorizationServicePort"

  e.g.2. % ./configure \
            --enable-gateway-mode \
            --with-globus-tcp-port-range=6201,6299 \
            --with-https-port=6288 \
            --with-eventlog-path=/data/bestman/log \
            --with-cachelog-path=/data/bestman/log \
            --with-certfile-path=/etc/grid-security/hostcert.pem \
            --with-keyfile-path=/etc/grid-security/hostkey.pem \
            --with-tokens-list="USATLASDATA1[desc:USATLASDATA1][owner:atlas][retention:REPLICA][latency:ONLINE][path:/data/atlas][12]" \
            --with-allowed-paths="/data/atlas;/tmp" \
            --enable-checksum-listing \
            --enable-sudofsmng \
            --enable-gums \
            --with-gums-url="https://gumsserver.fnal.gov:8443/gums/services/GUMSXACMLAuthorizationServicePort"

   e.g.3. % ./configure \
            --enable-gateway-mode \
            --with-globus-tcp-port-range=6201,6299 \
            --with-https-port=6288 \
            --with-eventlog-path=/data/bestman/log \
            --with-cachelog-path=/data/bestman/log \
            --with-certfile-path=/etc/grid-security/hostcert.pem \
            --with-keyfile-path=/etc/grid-security/hostkey.pem \
            --with-allowed-paths="/data/atlas;/tmp" \
            --enable-gums \
            --with-gums-url="https://gumsserver.lbl.gov:8443/gums/services/GUMSXACMLAuthorizationServicePort" \
            --with-tokens-list="USATLASDATA1[desc:USATLASDATA1][path:/data/atlas][120]" \
            --enable-sudofsmng \
            --with-extra-libs="-Dedu.unl.rcf.BestmanGridftpSelector.host=localhost -Dedu.unl.rcf.BestmanGridftpSelector.port=8649"

   e.g.4.  % ./configure \
           --enable-full-mode \
           --with-replica-storage-path=/data/bestman/cache \
           --with-replica-storage-size=2000 \
           --with-globus-tcp-port-range=6201,6299 \
           --with-https-port=6288 \
           --with-eventlog-path=/data2/bestman/log \
           --with-cachelog-path=/data2/bestman/log \
           --with-certfile-path=/etc/grid-security/srm/srmcert.pem \
           --with-keyfile-path=/etc/grid-security/srm/srmkey.pem

   e.g.5. % ./configure \
            --enable-gateway-mode \
            --with-globus-tcp-port-range=6201,6299 \
            --with-https-port=6288 \
            --with-eventlog-path=/data/asim/bestman/log \
            --with-cachelog-path=/data/asim/bestman/log \
            --with-certfile-path=/etc/grid-security/hostcert.pem \
            --with-keyfile-path=/etc/grid-security/hostkey.pem \
            --enable-gums \
            --with-gums-url="https://gumsserver.lbl.gov:8443/gums/services/GUMSXACMLAuthorizationServicePort" \
            --with-tokens-list="USATLASDATA1[desc:USATLASDATA1][owner:atlas][retention:REPLICA][latency:ONLINE][path:/data/atlas][120]" \
            --with-allowed-paths=/data/atlas \
            --enable-checksum-listing \
            --enable-sudofsmng

   e.g.6.  % ./configure \
           --enable-full-mode \
           --with-replica-storage-path=/data/bestman/cache \
           --with-replica-storage-size=2000 \
           --with-globus-tcp-port-range=6201,6299 \
           --with-https-port=6288 \
           --with-eventlog-path=/data2/bestman/log \
           --with-cachelog-path=/data2/bestman/log \
           --with-certfile-path=/etc/grid-security/srm/srmcert.pem \
           --with-keyfile-path=/etc/grid-security/srm/srmkey.pem \
           --enable-gums \
           --with-gums-url="https://gumsserver.lbl.gov:8443/gums/services/GUMSXACMLAuthorizationServicePort" \
           --with-globus-location=/software/globus-4.0.3

   e.g.7. % ./configure \
           --enable-gateway-mode \
           --with-globus-tcp-port-range=6201,6299 \
           --with-https-port=6288 \
           --with-eventlog-path=/data/bestman/log \
           --with-certfile-path=/etc/gridtest/srmcert.pem \
           --with-keyfile-path=/etc/gridtest/srmkey.pem \
           --with-transfer-servers="gsiftp://host1.domain.tld;gsiftp://host2;domain.tld" \
           --enable-gsiftpfsmng

   e.g.8.  % ./configure \
            --enable-gateway-mode \
            --with-globus-tcp-port-range=6201,6299 \
            --with-https-port=6288 \
            --with-eventlog-path=/data/bestman/log \
            --with-certfile-path=/etc/gridtest/srmcert.pem \
            --with-keyfile-path=/etc/gridtest/srmkey.pem \
            --enable-sudofsmng \
            --disable-pathfortoken \
            --with-tokens-list="DATA1[desc:USATLASDATA1][1000000];DATA2[desc:USATLASDATA2][1200000]"

   e.g.9.  % ./configure \
            --enable-gateway-mode \
            --with-globus-tcp-port-range=6201,6299 \
            --with-https-port=6288 \
            --with-eventlog-path=/data/bestman/log \
            --with-certfile-path=/etc/gridtest/srmcert.pem \
            --with-keyfile-path=/etc/gridtest/srmkey.pem \
            --enable-sudofsmng \
            --with-tokens-list="DATA1[desc:USATLASDATA1][1000000];DATA2[desc:USATLASDATA2][1200000]"

   e.g.10. % /configure \
           --enable-gateway-mode \
           --with-globus-tcp-port-range=6201,6299 \
           --with-https-port=6288 \
           --with-eventlog-path=/data/bestman/log \
           --with-cachelog-path=/data/bestman/log \
           --with-certfile-path=/etc/gridtest/srmcert.pem \
           --with-keyfile-path=/etc/gridtest/srmkey.pem \
           --with-blocked-paths=/home \
           --enable-sudofsmng \
           --enable-sudols

   e.g.11. % ./configure \
           --enable-gateway-mode \
           --with-globus-tcp-port-range=6201,6299 \
           --with-https-port=6288 \
           --with-eventlog-path=/data/bestman/log \
           --with-cachelog-path=/data/bestman/log \
           --with-certfile-path=/etc/gridtest/srmcert.pem \
           --with-keyfile-path=/etc/gridtest/srmkey.pem \
           --with-tokens-list="USATLASDATA1[desc:USATLASDATA1][owner:atlas][retention:REPLICA][latency:ONLINE][path:/data/atlas][120]" \
           --with-globus-location=/software/globus-4.0.3 \
           --enable-gsiftpfsmng \
           --with-allowed-paths=/data/atlas

   e.g.12. % ./configure \
            --enable-gateway-mode \
            --with-globus-tcp-port-range=6201,6299 \
            --with-https-port=6288 \
            --with-eventlog-path=/data/bestman/log \
            --with-cachelog-path=/data/bestman/log \
            --with-certfile-path=/etc/gridtest/srmcert.pem \
            --with-keyfile-path=/etc/gridtest/srmkey.pem \
            --enable-gsiftpfsmng \
            --with-tokens-list="data[desc:mydata][10];data2[desc:mydata2][12]" \
            --with-allowed-paths=/data/atlas

   e.g.13. % ./configure \
            --enable-clientonly \
            --with-globus-tcp-port-range=6201,6299

   e.g.14. % ./configure \
            --enable-testeronly \
            --with-globus-tcp-port-range=6201,6299

3. When user controlled storage spaces are accessible through BeStMan,
   --enable-sudofsmng would be good to have as one of configure options.
   It will enable sudo access to the user controlled storage space for BeStMan,
   so that mkdir, rmdir, rm, mv and cp can be executed under the user
   credential. When this option is used, /etc/sudoers file needs to be modified
   when non-root account runs BeStMan server.  Suppose "daemon" account
   runs BeStMan server, and the following additional entries are needed in
   /etc/sudoers file:

   Cmnd_Alias SRM_CMD = /bin/rm,/bin/mkdir,/bin/rmdir,/bin/mv,/bin/cp,/bin/ls
   Runas_Alias SRM_USR = ALL, !root
   daemon   ALL=(SRM_USR) NOPASSWD: SRM_CMD

   --enable-sudols is also available for ls to be executed under the user
   credential. Becuase sudo access is an expensive operation and ls is one of
   the frequently used operation, there can be a performance issue when it
   is turned on.

   An example of SURL in such case to access user managed storage space:
      srm://host:port/srm/v2/server\?SFN=/user_dir/user_path/user_file
   An example of SURL in BeStMan managed cache:
      srm://host:port/srmv2/server\?SFN=/srmcache/mapped_login/file_name


Note: More information can be found on http://sdm.lbl.gov/bestman

############################################################################
Note: When space reservation needs to be configured other than the default,
############################################################################

consider adding the following parameters in conf/bestman.rc after configure.
# for 5 GB quota allocation per user
ReplicaQualityStorageUserQuotaMB=5000
# for 40% of the storage that can be reserved
PublicSpaceProportion=60
# 1GB quota allocation per request
DefaultMBPerToken=1000


############################################################################
**** Note for scalability:
############################################################################

1. Adjust Sysctl values on the node where bestman server runs
Put these in /etc/sysctl.conf then run sysctl -p to apply them.
No need to reboot, now your kernel should be able to handle a lot more open connections.
Ref: http://fasterdata.es.net/tuning.html
     http://fasterdata.es.net/TCP-tuning/linux.html

# General gigabit tuning:
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_syncookies = 1
# this gives the kernel more memory for tcp
# which you need with many (100k+) open socket connections
net.ipv4.tcp_mem = 50576   64768   98152
net.core.netdev_max_backlog = 2500
# I was also masquerading the port comet was on, you might not need this
net.ipv4.netfilter.ip_conntrack_max = 1048576

In addition to above, check and changes these:
/sbin/sysctl -w net.core.somaxconn=256
/sbin/sysctl -w net.core.netdev_max_backlog=2500

Also, check these values for long timeout values.  If relatively short, keep them longer.
% sysctl \
net.ipv4.tcp_keepalive_time \
net.ipv4.tcp_keepalive_intvl \
net.ipv4.tcp_keepalive_probes

net.ipv4.tcp_keepalive_time = 1800
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_keepalive_probes = 9


2. Adjust Jetty web server control values
2.1 Increase the maximum container thread pool size for the jetty server
  --with-max-container-threads=<INT>
  e.g. configure --with-max-container-threads=1024

2.2 Have large size for jetty http connection acceptor threads
   --with-connection-acceptor-thread-size=<INT> Specify the number of acceptor threads available
                                 for the server's channel connector
  e.g. configure --with-connection-acceptor-thread-size=8

2.3 Have large size for jetty http connection waiting queue
   --with-connector-queue-size=<INT>   Specify the size of the http connector queue size
  e.g. configure --with-connector-queue-size=512


3. Increase the max java heap memory size
   --with-max-java-heap=<INT>
  e.g. configure --with-max-java-heap=4096

4. Client side http connection timeout control - bestman clients (only)
  e.g. srm-copy -sethttptimeout 1800

5. Each connection we make requires an ephemeral port, and thus a file descriptor,
and by default this is limited to 1024.
To avoid the Too many open files problem you'll need to modify the ulimit for your shell.

limit descriptors  65535
Or
unlimit

This can be changed in /etc/security/limits.conf, but requires a logout/login.
For now you can just sudo and modify the current shell
(su back to your non-priv'ed user after calling ulimit if you don't want to run as root):

$ sudo bash
# ulimit -n 999999
# erl

###################################################################
