v3_10_18
	Added GlideinOverloadEnabled to the activity logs for monitoring purposes (PR#629, Issue#569)
	Added custom scripts timeout option (GLIDEIN_CUSTOM_SCRIPTS_TIMEOUT, defaults to 600s) and cscripttrace option in GLIDEIN_DEBUG_OPTIONS to print the trace if they are shell scripts (PR#651)
	gettarballs.py now has an improved management of default tarballs (PR#627)
	Print triggered limits in Glidein requests in glideinFrontendElement (PR#653)
	Updated release script to the new OSG server and Git
	Added new startup scripts to prepare for better systemd compliance
	Bug fix: Fixed ownership of the /var/lib/gwms-factory in RPM package (PR#637)
	Bug fix: Fixing inconsistent use of string and bytes types breaking job monitoring (PR#650, Issue#649, Issue#646)
	Bug fix: Improved cvmfs_helper_funcs.sh to be faster and have consistent names (PR#651)
	Bug fix: OSG_autoconf, handle missing pilot entries gracefully in merge_yaml (PR#655, Issue#654)
	Bug fix: OSG_autoconf, fix missing GPU count for whole-node GPU entries. (PR#657, Issue#656)
	NOTE: Custom scripts execution times out after 600s and scripts are killed with SIGTERM

v3_10_17
	Recognize EL/CentOS 10 worker nodes to select the correct HTCondor tarball (PR#600)
	Prevented memory spikes during Factory reconfiguration by copying Condor tarballs instead of loading them into memory (PR#602, Issue#601)
	Reuse Condor tarballs across reconfigurations by generating hash-based filenames from file metadata instead of timestamps (PR#604, Issue#603)
	Factory monitoring now showing Client Requested Idle Glideins; only keeping track of Factory adjusted Idle (PR#606, Issue#520)
	Added support for HTCondor Python bindings v2. If available, v1 is still preferred (PR#608)
	Removed ownership and HTCondor checks connected to GWMS 3.5 migration (PR#608)
	Added a reconfig/upgrade warning when user job wrapper scripts require more than sh or contain an exec statement (Issue#584, PR#610, PR#618)
	Added a word checker to warn about misspelling of attributes in the configuration files (PR#616)
	Removed dependency from SysVInit in spec file and package installation. Using native Systemd (PR#617)
	Improved Frontend docstrings (PR#574)
	Bug fix: Removed coral frontend from first page animation (PR#605)
	Bug fix: Fixed upgrade crashes in RRD verifier due to improper NamedTemporaryFile handling (Issue#619, PR#620)
	NOTE: Job wrapper scripts must be "POSIX" compatible. Bash is enabled when available but not guaranteed. Also, scripts should not include `exec`. GlideinWMS will print a warning during reconfiguration.
	NOTE: If you have the Python library Jellyfish (pip install jellyfish), a new word checker will warn you if you misspell a configuration attribute.

v3_10_16
	create_cvmfsexec_distros.sh updated adding EL9 to platforms and improved command syntax (Issue#549, PR#582)
	Added new staticextra resource type – behaves like static (creates one dedicated slot per instance with a virtual CPU each), but instead of subtracting memory from the main partitionable slot, it adds memory to it (Issue#590, PR#591)
	OSG_autoconf: Added mapping for EIC and CLAS12 and support for the new dedicated `OSG` collector parameter to set `EstimatedCPUs` replacing the use of the generic `CPUs` (PR#595)
	Bug fix: Fixed incorrect glog_get_logfile_path_relative call in condor_startup.sh (PR#579)
	Bug fix: Fixed Frontend reconfiguration failure when using "ALL" schedd (Issue#575, PR#580)
	Bug fix: Fixed cvmfsexec failure when duplicate repositories are in the GLIDEIN_CVMFS_REPOS list (Issue#567, PR#585)
	Bug fix: Fixed wrong exec command in wrapper script (Issue#584, PR#587)
	Bug fix: Environment cleanup fixed. `PATH`, `LD_LIBRARY_PATH`, `PYTHONPATH`, and `LD_PRELOAD` are now correctly cleared when requested with `clearpaths` or `clearall` (Issue#592, PR#596)
	Bug fix: Fixed maxWallTime not being set as a submit attribute by OSG_autoconf when an entry is whole node (PR#595)
	NOTE: Added httpd configuration to enhance security by disabling version headers and trace (PR#578)
	NOTE: Since 3.10.15, all job wrapper scripts must be "POSIX" compatible to be able to run on small images with Busybox. Added a relaxation in 3.10.16, setting `set +o posix` when Bash is available (PR#587)

v3_10_15
	Added GLIDEIN_OVERLOAD_ENABLED to control partial CPU and memory overload (PR#536)
	Added GLIDEIN_CVMFS_REPOS custom variable to define additional CVMFS repositories to mount (PR#547)
	Added ramp_up_attenuation config parameter to control Glidein provisioning remap-up (PR#556)
	Updates the pilot generation logic in OSG_autoconf to check the OSG_BatchSystem attribute from the OSG collector. If the batch system is set to "CONDOR", the resulting pilot entry will have work_dir set to "Condor" (PR#558)
	Updates the pilot generation logic in OSG_autoconf to use the cpus attribute from the OSG collector to set GLIDEIN_ESTIMATED_CPUs (PR#560)
	Bug fix: Retrieve and use CVMFS_MOUNT_DIR from glidein_config if not in the environment (PR#552)
	Bug fix: Addressed POSIX mode problems and CVMFS path resolution (PR#555)
	NOTE: The job wrappers in the Glidein are now running with the `/bin/sh` prompt instead of `/bin/bash`. They use Bash and `set +o posix` when possible, but there may be another shell.
	NOTE: Duplicate repositories in the GLIDEIN_CVMFS_REPOS list will fail cvmfsexec.

v3_10_14
	Added support for Ubuntu 24 workers (PR#529)
	Add keyword ALL to query all schedulers known to the collector without listing them explicitly (Issue#510, PR#532)
	Updated download/execution order of `cvmfs_setup.sh` during glidein startup using a new priority `precvmfs_file_list` (PR#528)
	Add keyword usertrace to the GLIDEIN_DEBUG_OPTIONS custom variable to enable shell tracing in user jobs and wrapper (PR#540)
	Glideins can start containers with Busybox and no Bash, e.g Alpine Linux. The Glidein itself still requires Bash (Issue#538, PR#540)
	Bug fix: Removed incorrect warning when setting SINGULARITY_BIN to keyword (PR#534)
	Bug fix: Added `--skip-broken` to yumalldeps to avoid an error when incompatible packages are in the list (Issue#530, PR#534)
	Bug fix: Added explicit retrieval from glidein_config of GLIDEIN_CONTAINER_ENV and GLIDEIN_CONTAINER_ENV_CLEARLIST in singularity_lib.sh (PR#535)
	Bug fix: Fixed handling of Apptainer environment and image restrictions (PR#535, PR#539)
	Bug fix: Added workaround for HTCondor setting PATH only as variable and not in the environment (PR#539)
	NOTE: In the Glidein, `cvmfs_setup.sh` is now executing earlier, right after entry scripts (PR#528)
	NOTE: Removed compatibility with GWMS<3.4.5. GlideinWMS v3.4.6 up to v3.10.14 Factories and Frontend are interoperable.

v3_10_13
	Bug fix: Fixed a credential rewriting error caused by PR#511 and a path error in PR#515 (PR#519)

v3_10_12
	Exporting GLIDEIN_Name and GLIDEIN_UUID to the Glidein environment, for all scripts running inside the Glidein (PR#512)
	HTCondor LOCAL_CONFIG_DIR support for the Glidein HTCondor daemons (PR#515)
	Improved docstrings in the Factory module (PR#511)
	Bug fix: Fixed misleading counts related to the number of jobs that run in the Glidein (Issue#456, PR#516)

v3_10_11
	Added a test Apptainer image to use when the configured one is not available (PR#482)
	Added a new configuration knob, stale_age, for Factory entries to control the age of the Glideins to be considered stale for certain statuses (PR#494)
	Update get_tarballs to use new directory called beta (PR#495)
	Support GPUs in the mapping of OSG_autoconf VOs (PR#496)
	Made the Frontend library more friendly to other clients, e.g. Decision Engine (PR#504)
	Bug fix: Protect processing of custom scripts in glidein_startup.sh against stdin interference (PR#498, Issue#500)
	Bug fix: Some config files used in the RPM package, including the httpd ones, were obsolete and not the version in the source tree. (PR#492, PR#502)
	Bug fix: Host IP is now searched in blacklist also when the host command is missing (PR#499, Issue#493)
	Bug fix: Added missing HTCondor requirements in spec file (PR#502)
	Bug fix: Unset CONDOR_INHERIT before condor startup to avoid any conflict in the condor configurations (PR#503, Issue#274)
	NOTE: Monitoring pages are now redirecting to https if available, i.e. mod_ssl is installed and mod_ssl.conf is present. This behavior was present in the past but had been lost and now it has been reinstated. (PR#492, PR#502)
	NOTE-FRONTEND: The default Frontend tokens key is now variable, $HOME/passwords.d/UPPERCASE_USERNAME. There is no actual change since this is /var/lib/gwms-frontend/passwords.d/FRONTEND for normal RPM installations. (PR#504)
	NOTE-FACTORY: The new variable SINGULARITY_IMAGE_REQUIRED defaults to false and allows to use Singularity/Apptainer also when the configured image is not available. The image must be provided by the job or a future custom script in order not to fail. (PR#482)
	NOTE-FACTORY: APPTAINER_TEST_IMAGE can be set to an always available Singularity/Apptainer image to use for testing. Defaults to oras://ghcr.io/apptainer/alpine:latest (PR#482)

v3_10_10
	Improved gfdiff tool interface and changed diff algorithm (PR#476)
	Added option to check if new HTCondor tarballs are available to get_tarballs (PR#477)
	Bug fix: Fixed gfdiff issue with mergely interface (PR#476)
	Bug fix: Fallback to cp/chown when cp -p does not work (It is unavailable on some containers) (PR#478)

v3_10_9
	Added custom JWT-authenticated log server example (new RPM glideinwms-logging) (Issue#398, PR#467)
	Now using also Apptainer included in the HTCondor tar ball (Issue#364, PR#473)
	Improved the docstrings and some code in the lib files and few others with the help of AI (PR#471, PR#472)
	Added --skip-rpm option in release.py to skip RPM building (PR#474)
	Bug fix: Fixed early truncation in log files configuration and inconsistent documentation (Issue#464, PR#462, PR#463)
	Bug fix: Removed confusing tac broken pipe messages from the Glidein stderr (PR#465)
	Bug fix: Fixed JWT logging credentials not transferred to the Glidein. This includes removal of DictFile.append() and use of add_environment() for JWT tokens (Issue#398, PR#467)
	Bug fix: Fixed quotes in Glidein command line unpacking and replaced deprecated add_config_line commands (PR#468)
	Bug fix: Allow anonymous SSL authentication for the dynamically generated client config (Issue#222, PR#470)
	Bug fix: Checking also the apptainer binary in the SINGULARITY_BIN path, not only singularity (PR#473)
	NOTE: The Glidein will always send SIGQUIT to HTCondor when it receives INT, TERM, QUIT signals. This speeds up the shutdown (PR#466)
	NOTE: Renamed Glidein Logging functions to glog_... (glog_init, glog_setup, glog_write, glog_send - PR#467). The original functions (log_init_tokens, log_init, log_setup, log_write, send_logs_to_remote, ...) are no more available
	NOTE: Apptainer downloaded in the HTCondor tar ball is now considered after the PATH in the search for a valid binary. The keyword CONDOR in the parameter SINGULARITY_BIN will make so that the HTCondor Apptainer is preferred ahead of the rest (PR#473)
	NOTE: Added RHEL9 to the list of default OSes used for the container images lookup. Now it is default,rhel9,rhel7,rhel6,rhel8 (PR#473)

v3_10_8
	Advertising information about unprivileged user namespaces in glidein classad (PR#416)
	Added option --group-name option to manual_glidein_submit (PR#435)
	Bug fix: Fixed root unable to remove other users jobs in the Factory (PR#433)
	Bug fix: HTCondor TRUST_DOMAIN configuration macro set to string to avoid Glidein config error (PR#420)
	Bug fix: Disabled shebang mangling in rpm_build to avoid gwms-python not finding the shell (Issue#436, PR#437)
	Bug fix: Dynamic creation of HTCondor IDTOKEN password (Issue#440, PR#441)
	Bug fix: Autodetect CONDOR_OS in the manual_glidein_submit tool (Issue#449, PR#453)
	Bug fix: Failed log rotation due to wrong file creation time (Issue#451, PR#457)
	Bug fix: Replacing xmlrunner with unittest-xml-reporting (PR#428)
	Bug fix: Updated the release upload script to work with osg-sw-submit (PR#439)

v3_10_7
	Apptainer cache and temporary directory set in the Glidein working directory (Issue#403, PR#404)
	Ability to set a minimum required memory for partitionable Glideins. The default is the value used previously, 2500 MB (Issue#405, PR#406)
	Blackhole Detection. Stop accepting jobs if they are consumed at a rate higher than the configured limit and declare the Glidein a blackhole (Issue#331, PR#399)
	Bug fix: Fixed Apptainer validation not considering uid_map w/o initial blank (Issue#395, PR#396)
	Bug fix: Flattening error message in \_CONDOR_WRAPPER_ERROR_FILE and JobWrapperFailure Ad. This is a workaround for a bug in HTCondor (PR#400)
	Bug fix: Fixed problem when check_signature in glidein_startup is not defined (PR#402)
	Bug fix: get_tarballs look for HTCondor releases also in the update directory (Issue#412, PR#413)

v3_10_6
	Add knobs to allow overloading of memory, GLIDEIN_OVERLOAD_MEMORY, and CPU, GLIDEIN_OVERLOAD_CPUS. (Issue#370, PR#374)
	Added HTCondor tarball downloader (Issue#367, PR#366)
	Added default (/bin,/usr/bin) when PATH is empty in glidein_startup.sh (PR#373)
	Advertising Factory's HTCondor submit parameters (Issue#307, PR#382)
	Bug fix: Changed M2Crypto imports to be compatible with 0.40.0 the code must import also the components (PR#377)
	Bug fix: Fixed PATHs handling in glidein_startup.sh (PR#379)
	Bug fix: Fixed match policy_file import failure (Issue#378, PR#380)
	Bug fix: Fixed syntax error in ClassAd used for gangliad configuration (Issue#368, PR#385)
	Bug fix: Added extra logging to investigate file rotation problem (Issue#362, PR#389)
	Bug fix: Fixed writing of missing dict files during upgrade (Issue#388, PR#391)
	NOTE-FACTORY: The submit attributes (submit/submit_attrs) are now published in the glidefactory classad with the GlideinSubmit prefix followed by the attribute name and same value. If the attribute name starts with "+" this will be replaced by "_PLUS_", since only alphanumeric characters and "_" are valid in ClassAd attribute names.

v3_10_5
	Bug fix: Workaround for EL7 PyJWT bug, generating bytes instead of str (PR#355)
	Bug fix: Fixed missing `cvmfsexec.cfg` files from Factory reconfig and improved cvmfsexec warnings (Issue#348, PR#356)
	Bug fix: Added bash requirement to files using bashisms, notably `glidein_sitewms_setup.sh` (PR#358)
	Bug fix: Fixed syntax errors in analyze_queues (PR#357)
	Bug fix: Fixed setup_x509 to be successful w/ TRUST_DOMAIN set in the as Factory or Frontend parameter (PR#359)
	Bug fix: GLIDEIN_SINGULARITY_BINARY_OVERRIDE set also with Frontend and Factory params, not only WN environment (PR#360)

v3_10_4
	Bug fix: Fixed missing arguments from rrdtool fetch call (Issue#351, PR#352)
	Bug fix: gconfig.py to use `gwms-python`, not use `python3` (Issue#349, PR#350)
	Bug fix: Fixed alternative Shell code still using the `python` (i.e. python2) interpreter (Issue#289, PR#353)

v3_10_3
	Added support for Debian 11 and Ubuntu 22 worker nodes (PR#320)
	Added structured logging. It is a hybrid format with some fields followed by a JSON dictionary. The exact format of the messages may change in the future, and we plan for it to become the default. Now it is disabled by default. Add `structured="True"` to all `process_log` elements (PR#333)
	Add option to set xml output directory in OSG_autoconf (PR#319)
	Allow OSG_autoconf to skip sites or CEs that are not present in the OSG collector (PR#315)
	Add option to set num_factories in OSG_autoconf (Issue#344, PR#345)
	Added the ability to clear a list of variables from the environment via GLIDEIN_CONTAINER_ENV_CLEARLIST before starting a container (Issue#341, PR#342)
	Bug fix: Removed `classad` from requirements.txt. The HTCSS team distributes only the `htcondor` library in PyPI which includes a different version of classad (PR#301)
	Bug fix: Fixing Python 3.9 deprecations (`imp`, `getchildren()` in `xml.etree.ElementTree`) (PR#302, PR#303)
	Bug fix: Populate missing Entry parameters for ARC CEs submissions (PR#304)
	Bug fix: Modified the usage of subprocess module, for building/rebuilding cvmfsexec distributions, only when necessary (PR#309)
	Bug fix: manual_glidein_submit now correctly sets idtokens in the EncryptedInputFiles (issue#283, PR#284)
	Bug fix: Fixed fetch_rrd crash in EL9 causing missing monitoring and glidefactoryclient classad information (Issue#338, PR#339)
	NOTE-FACTORY: When generating cvmfsexec distribution for EL9 machine type on an EL7 machine, the factory reconfig and/or upgrade fails as a result of an error in `create_cvmfsexec_distros.sh`. This is possibly due to the tools for EL7 being unable to handle EL9 files (as per Dave Dykstra). Please exercise caution if using `rhel9-x86_64` in the `mtypes` list for the `cvmfsexec_distro` tag in factory configuration. - Our workaround is to remove the EL9 machine type from the default list of machine types supported by the custom distros creation script.  Add it back if you are running on an EL9 system and want an EL9 cvmfsexec distrinution. (PR#312)'

v3_10_2
	Add a parameter to control the name of the keyname for idtokens (PR#268)
	Added a factory knob to allow control over rebuilding of cvmfsexec distributions (PR#272)
	RHEL9 worker node are now recognized by condor_platform_select automatic OS detection (PR#285)
	Removed pre-reconfigure hook used for rebuilding cvmfsexec distributions whenever a factory reconfig/upgrade was run (issue#262)
	Bug fix: Use correct variable for `$exit_code` in `singularity_exec_simple` (PR#259)
	Bug fix: Make sure default value does not overwrite the global one for idtoken_lifetime (PR#261)
	Bug fix: Protect OSG_autoconf from OSG collector unavailability (PR#276)
	Bug fix: Fixed jobs going in unknown state in factory monitoring. added QUEUING state for new ARC-CEs REST (PR#286)
	NOTE: When using on-demand CVMFS, all Glideins after the first one on a node are failing (issue#287)
	NOTE-FACTORY: The rebuilding of cvmfsexec distributions is now disabled by default, it must be enabled via the new factory knob.\ If you are upgrading from v3.9.5 or later, check that the cvmfsexec script is no more in the pre-reconfigure hooks directory.

v3_10_1
	Added utility function to replace error_gen in python scripts (PR#254)
	Bug fix: Fixed status reporting for `gconfig.py`  (PR#254)
	NOTE: Please see all the notes in 3.9.6 about the custom script changes

v3_10_0
	All the features and fixes in v3_9_6
	Use `SINGULARITY_DISABLE_PID_NAMESPACES` disable `--pid` in Singularity/Apptainer  (OSG SOFTWARE-5340, PR#232)
	Raise a warning if there are "FATAL" errors in Singularity/Apptainer stdlog and the exit code is 0 (PR#235)
	Added `gconfig.py`. Python utilities to read and write glidein_config (PR#237)
	Bug fix: Set PATH to default value instead of emptying it (PR#233)
	Bug fix: Have `get_prop_str` return the default value when the attribute is "undefined" and `gwms_from_config` return the default when set instead of an empty string (PR#235)
	Bug fix: Fixed credential ID in Glideins. It was not set for scitokens causing incorrect monitoring values (PR#242)
	Bug fix: Fixed typo in `singularity_lib.sh` (issue#249)
	NOTE: This follows v3_9_6. Please see all the notes about the custom script changes

v3_9_6
	Added token authentication to Glideins running in the Cloud (AWS and GCE). Now Glideins on all supported resources can authenticate back using IDTOKENS when using recent HTCSS versions.
	Added `GLIDEIN_PERIODIC_SCRIPT` env variable to notify periodic custom scripts
	Added the possibility to set the Glidein HTCSS TRUST_DOMAIN as attribute in the Frontend configuration
	The Frontend configuration is now valid (reconfig/upgrade successful) even if some HTCSS schedds are not in DNS. Failing only if all schedds are unknown to DNS
	Bug fix: Fixed `glidien_config` corrupted by concurrent custom scripts run via HTCSS startd cron (#163)
	Bug fix: Fixed `setup_x509.sh` not to write to stdout when running as periodic script in HTCSS start cron (issues#162,#164)
	Bug fix: Fixed setup_x509.sh creates proxy file in directory used for tokens (issue#201)
	Bug fix: Fixed GLIDEIN_START_DIR_ORIG and GLIDEIN_WORKSPACE_ORIG values in glidein_config
	Bug fix: Fixed unnecessary proxy/hostcert.pem workaround in frontend config (issue#66)
	Bug fix: Fixed analyze_entries and python3 readiness (issue#194)
	Bug fix: Fixed gwms-renew-proxies service should check if local VOMS cert is expired (issue#21)
	Bug fix: Fixed python3 check return value in case of exception (PR#211)
	Bug fix: Fixed list_get_intersection in singularity_lib.sh that was requiring python2 (PR#212)
	Bug fix: Unset SEC_PASSWORD_DIRECTORY in the Glidein HTCSS configuration, was causing warnings for unknown files (PR#226)
	Bug fix: HTCSS DC_DAEMON_LIST now is equal to DAEMON_LIST only in the Factory, in all other GlideinWMS components only selected HTCSS daemons are added explicitly to it (issue#205)
	Bug fix: Working and local tmp directories are removed during Glidein cleanup also when the start directory is missing. This result in a loss of Glidein final status information but avoids sandbox leaks on the Worker Node. (issue#189)
	NOTE: custom scripts should always read values via gconfig_get(). The only exception is the parsing or the line to get the add_config_line source file
	NOTE: add_config_line is deprecated in favor of gconfig_add. Will be removed from future versions
	NOTE: add_config_line_safe is deprecated in favor of gconfig_add_safe. gconfig_add is the recommended method to use also in concurrent scripts
	NOTE: gconfig_add_unsafe, with a similar behavior to the former add_config_line, has been added. Its use is not recommended, unless absolutely sure that no script will be run in parallel
	NOTE: Only the first collector in TRUST_DOMAIN is kept, following collectors are removed. This happens both in the Frontend token issuer and in the setting of the Glidein TRUST_DOMAIN (setup_x509.sh).

v3_9_5
	Support for credential generator plugins
	Support for Apptainer
	Support for python3-rrdtool added in OSG 3.6 also for RHEL7
	Support for CVMFS on-demand via cvmfsexec (mount/umountrepo)
	Configurable idtokens lifetime
	Improved frontend logging
	Improved default SHARED_PORT configuration
	Special handling of multiline condor config values
	Advertise CONTAINERSW_ variables
	Bug fix: Fixes an issue causing SciTokens transferred to the factory to be corrupted
	Bug fix: Fixes to manual_glidein_submit for compatibility with Python 3
	Bug fix: Fixes to OSG_autoconf for compatibility with Python 3
	Bug fix: Fixes a minor issue in job.condor
	Bug fix: Fixes OS detection for CentOS Stream 8
	Bug fix: Fixes an issue with scitoken handling to gsi_proxy configured factory entries
	Bug fix: Fixes condor tarball directory structure
	Bug fix: Stops clearing LD_LIBRARY_PATH before starting HTCondor
	Bug fix: Ensures that idtokens are transferred to glidein credential directory

v3_9_4
	Includes all features and fixes of 3.7.6
	Adds support for CentOS 8
	Adds two new RPM packages (vofrontend-libs and vofrontend-glidein) to provide essential libraries for frontend-like applications
	Updates license to Apache 2.0
	Bug fix: Updates libraries for compatibility with PyYAML 6.0
	Bug fix: Fixes a string encoding issue affecting the cat log scripts
	Bug fix: Fixes AWS/Google metadata encoding error

v3_9_3
	Includes all features and fixes of 3.7.5
	Adds type validation to the check_python3_expr.py script
	Removes unused encodingSupport.py and its unit tests
	Bug fix: Fixes an encoding issue affecting cloud submissions

v3_9_2
	Includes all features and fixes of 3.7.3 and 3.7.4
	Drops support for BDII along with its LDAP requirements
	Adds check_python3_expr.py to validate python3 in match expressions
	Bug fix: Fixes string encoding issues on glideFactoryLib
	Bug fix: Fixes an issue that would cause some factory configurations to generate malformed Condor files

v3_9_1
	Includes all features and fixes of 3.7.2
	Separates httpd packages from Factory and Frontend RPMs
	Bug fix: Frontend no longer reports duplicate glideclient daemons to the Factory

v3_9
	Includes all features and fixes of 3.6.3 and 3.7.1
	Python 3 code, some pitfalls to pay attention to are in the Bug Fix section
	Bug fix: custom dictionaries with __getattr__ in the creation package (an infinite loop was masked in Python 2)
	Bug fix: str vs bytes, especially in crypto libraries
	Bug fix: None cannot be used in sorted string lists and we use (None, None, None) for unmatched jobs in the Frontend
	Bug fix: Add __repr__ method to SubParams class
	Bug fix: Fixed pycache of creation/lib included in multiple packages
	Bug fix: Fixed spec file to include python3 compiled files and not python2 ones
	Bug fix: Fixed Mapping import
	Bug fix: Fixed str vs bytes confusion in cryptography functions and classes

v3_7_7
	Authenticate w/ tokens w/ the HTCondor-CE collector
	Added CONTINUE_IF_NO_PROXY attribute to control GSI to token transition
	Bug fix: Fixes an issue causing SciTokens transferred to the factory to be corrupted
	Bug fix: Fixes condor_config -dump is not safe if the pool config has multiline entries
	Bug fix: Add job.condor typo fix for bosco to 3.7.7
	NOTE: Now branch_v3_7 is completely separate. Feature or fixes have to be backported

v3_7_6
	Flexible mount point for CVMFS in glidein
	Frontend and factory check expiration of SciTokens
	Shared port is default for User Collector
	API to retrieve job information from condor jobID
	OSG_autoconf now allows override yaml configuration files
	expiration and other validity of JWT tokens checked before use
	Bug Fix: condor_chirp no longer fail when python2 used in job
	Bug Fix: IDTOKEN issuer changed from collector host to trust domain
	Bug Fix: proxy was failing to renew when both SCITOKEN and grid_proxy auth used
	Bug Fix: OSG_autoconf automatic assignment of submit attributes fixed
	Bug Fix: OSG_autoconf common entry fields implemented
	Bug Fix: OSG_autoconf now possible to append instead of replace values
	Bug Fix: glidein clears LD_LIBRARY_PATH prior to starting HTCondor
	Bug Fix: JOB_TOKENS no longer set in glidein environment if not needed
	Bug Fix: IDTOKEN/SCITOKEN auth to bosco sites fixed
	Bug Fix: improved IDTOKEN revocation procedures
	Bug Fix: SSL added to condor *_AUTHENTICATION_METHODS
	Bug Fix: Possible to have frontend group with only SCITOKEN authentication
	Bug Fix: Factory stores frontend group information per-token
	Bug Fix: Intermittent -check_v35 error fails to  startup factory
	Bug Fix: Fix glidein_starup.sh in staging area to be one with payload
	Bug Fix: GCB support removed
	Bug Fix: glidein_cleanup can cleanup directories chmoded read-only
	Bug Fix: improved shellcheck testing
	Bug Fix: corrected paths for some variables in glideins
	Bug Fix: eliminated spurious log messages in glideins
	Bug Fix: STARTD_JOB_ATTRS fixed in glideins
	Bug Fix: GWMS_PATH in glidein_startup.sh corrected
	Bug Fix: better cleanup of tmpfiles after install

v3_7_5
	Support for big files in the glideinwms repository
	Additional download mechanism for files
	Bug Fix: Have SciToken credential be per-entry
	Bug Fix: Fix IDTOKEN generation in the Frontend
	Bug Fix: Cleanup script not executing correctly at glidein termination
	Bug Fix: Automount of CVMFS integrated into glidein script

v3_7_4
	Consider not available (not Idle) partitionable slots where all GPUs are used
	Bug Fix: Submissions from some 3.6.5 frontends to 3.7.3 factories go on hold

v3_7_3
	More efficient generation of IDTOKENS by frontend
	SciToken credentials configured and treated same as other credentials for glidein submission
	Glideins configured without GSI but with SciToken and IDTOKENS successfully run jobs.
	Order of places glidein checks for singularity changed
	PATH with multiple directories can be configured to override singularity binary
	Remove LD_PRELOAD from glidein environment by default
	Fail without fall-back when user provided singularity image missing
	Improved GPU implementation in glidenins
	OSG_autoconf more configurable, auth_method and pilot_entries can be set
	Bug Fix: Missing GSI configuration does not stop schedd query when IDTOKENS configured.
	Bug Fix: Factory should not check condor tarball version prior to SciToken authentication to CE
	Bug Fix: tmpfile needs to be on correct file system when factory writes tokens from frontend
	Bug Fix: IDTOKENS auth does not fail on improper condor_version comparison
	Bug Fix: rpm sub package (http) building issues fixed

v3_7_2
	Includes all features and fixes of 3.6.6
	Drop GLExec support
	Mechanism for running custom scripts at the end of glidein execution
	OSG_autoconf protection added to prevent failures when CE disappears from OSG Collector
	Bug fix: Frontend no longer over-generates tokens for entries
	Bug fix: Factory does not remove glideins when idle limit is hi

v3_7_1
	Includes all features and fixes of 3.6.5
	Configuration changes to make more compatible with HTCondor 8.9  on initial install
	SciTokens authentication between Factory and CE
	IDTokens authentication between Factory and Frontend
	Bug fix: Factory reconfigs and startups do not require manual creation of directories
	Bug fix: Factory builds Condor Tarballs to send with glideins correctly

v3_7
	Includes all features and fixes of 3.6.2
	Use of HTCondor token-auth for Glideins authentication
	Added secure logging channel
	Refactored glidien_startup.sh to separate out the code in heredoc sections and improve code quality

v3_6_5
	Added GitHub actions to test the code (Python unit tests, BARS, pylint, pycodestyle)
	Bug fix: Fix for paths in Singularity not substituted correctly when the execute directory has a link or bind mount
	Bug fix: Fixed shared_port_port to adapt to HTCondor 8.8.10 changes
	Bug fix: Fixed overrides in OSG_autoconf using depcopy, use https for gfdiff
	Bug fix: Fixed credential dir in manual_glidein_submit removing hardcoding

v3_6_4
	Improved OSG_autoconf
	Dropped Python 2.6 support
	Added support for RHEL 8 worker nodes
	Harmonized, improved and extended CI tests (same CLI, support for Python 3, ...)
	Added tool to verify if the code for a ticket has been merged
	Bug fix: Fix for condor_chirp not finding htchirp in the PYTHONPATH
	Bug fix: Fixed unquoted SINGULARITY_PATH causing HTCondor not to start

v3_6_3
	Add Python condor_chirp to PATH
	Dropped tarball installation support
	Add Python condor_chirp and other GWMS tools to PATH in Singularity
	Removed "Group:System Environment/Daemons" from glideinwms.spec in OSG repo
	Added new API documentation using sphinx
	Added a runner to run the different tests with consistent CLI
	Added new RHEL 8 Singularity image
	Updated the singularity wrapper with new features form the OSG wrapper
	Clean PYTHONPATH and setup GWMS environment in Singularity containers
	Improved the environment handling of the Singularity wrapper
	Bug fix: Upgrade the factory documentation not to ask to install the switchboard
	Bug fix: The Python version of condor_chirp now works with Singularity jobs
	Bug fix: Enable RFC proxies by default for direct proxies
	Bug fix: Fix OSG_autoconf and gfdiff
	Bug fix: Added COMPLETED to the known list of "GridJobStatus"es
	Bug fix: Add a knob to set entry limits in OSG_autoconf
	Bug fix: Stop pilots even after the downtime timestamp has been reached when draining pilots
	Bug fix: Glideins are unrecoverable unless listed as recoverable
	Bug fix: User job invocation via singularity eating up -e or -n

v3_6_2
	Add a portable condor_chirp for user jobs running in the glideins
	Automate the generation of factory configuration
	Improve control of setup of module and spack in Singularity using MODULE_USE
	Adding GPU monitor as default for pilots
	Reduce number of queries from Frontend to User collector
	Make lgwms-logcat.sh work with non standard factory instance names
	Bug fix: Pilot using an expired proxy when authenticating to the collector
	Bug fix: CI issues (coverage broken in SL6, script selection affected by file name)
	Bug fix: Incorrect CERTIFICATE_MAPFILE used when using HTCondor python binding (failed schedd authentication)
	Bug fix: Fix release manager relative imports
	Bug fix: Broken submission to GCE and AWS
	Bug fix: Correct how entry_ls detect an rpm installation
	Bug fix: Adjust size of ClusterId and ProcessId fields for the condor_q custom format
	Bug fix: Total of frontend info is 0 in Factory monitoring

v3_6_1
	Added compatibility w/ HTCondor 8.8.x in OSG 3.5
	Monitoring pages use https if available
	Improved search and testing of Singularity binary
	Unset LD_LIBRARY_PATH and PATH for jobs in Singularity
	Updated documentation links and Google search
	Improved CI testing
	Stop considering held limits when counting maximum jobs in Factory
	Bug fix: Fix Fatcory tools (entry_rm, entry_q and entry_ls) to be more verbose with single user Factory
	Bug fix: Removed hardcoded CVMFS requirement for Singularity
	Bug fix: Improve diagnostic messages when rsa.key file is corrupted
	Bug fix: Improve control of the worker node draining, avoiding to stop accepting jobs as soon as the draining file is present

v3_6
	Renaming of v3_5_1 (released on 9/18) to respect the odd/even numbering
	Includes all 3.4.6 features
	Updated SW and docs for the change in OSG factories
	Updated all the obsolete links to HTCondor manual in GlideinWMS website
	Set up an ITB Frontend for GWMS and FIFE testing
	Updated gitattributes to resolve conflicts w/ checksum files
	Added editorconfig and default encoding
	GlideinWMS code will use now Google docstring format
	Advertise if a Glidein can use privileged or unprivileged Singularity
	Check if single user factory migration script has been run before startup
	Bug fix: pip errors in nightly CI
	Bug fix: Unittest failing at times on SL7
	Bug fix: Factory could start also w/ a GT2 entry enabled
	NOTE: v3_6 is fully interoperable w/ v3_5 and v3_4. I.e. You can mix v3_6_x, v3_5_x and v3_4_x Factories and Frontends in your system
	NOTE-FACTORY: The v3.6 Factory (like v3.5) is using a single user. This is a major change. If you are upgrading, after the RPM upgrade you will need to:\ 1. stop Factory and HTCondor, 2. Migrate the HTCondor status running the fact_chown script, 3. Run upgrade and restart the HTCondor and the Factory. For detailed instructions see http://glideinwms.fnal.gov/doc.dev/factory/configuration.html#single_user

v3_5    Removed support for Globus GRAM (GT2/GT5)
        Removed dependency on condor_root_switchboard
        Improved release scripts release.py and osg-release.sh (better help and error messages)
        Added unit tests
        Improved Frontend scalability (from CMS scale tests)
        Remove use of deprecated sets module
        Bug fix: Incorrect exception handling glideinFrontendElement.py::populate_pubkey
        Bug fix: Entries in downtime setting seems not to work correctly
        Bug fix: GlideinWMS proxy renewal service broken for Xenon
        NOTE: v3_5 and v3_4 are fully interoperable. I.e. You can mic v3_5 and v3_4_x Factories and Frontends in your system
        NOTE-FACTORY: The v3.5 Factory is using a single user. This is a major change. If you are upgrading, after the RPM upgrade you will need to:
          1. stop Factory and HTCondor, 2. Migrate the HTCondor status running the fact_chown script, 3. Run upgrade and restart the HTCondor and the Factory.
          For detailed instructions see: http://glideinwms.fnal.gov/doc.dev/factory/configuration.html#single_user

v3_4_6  Make Factory compatible with older 3.4 Frontends reverting back to send voms and glexec attributes as strings (Fermilab Frontend not communicating w/ upgraded Factory)
        Backport: GlideinWMS proxy renewal service broken for Xenon
        Add always --contain to the Singularity invocation and update wrapper adding improvement in the OSG one
        Document and expand multi-node Glidein
        Document usage of ignore_down_entries
        Bug fix: Not escaped comma in GSI_DAEMON_NAME causing problems
        Bug fix: Debug options causing Singularity invocation to fail
        Bug fix: Singularity workdir creation failing at some sites
        Bug fix: Fixing chkconfig lines on proxy renewal (SOFTWARE-3758)

v3_4_5  Add a bind mount for /etc/localtime in Singularity
        Bug fix: Strings evaluated as boolean (New error preventing the Frontend to match jobs)

v3_4_4  Propagate to Factory and glidein submission attributes controlled by FE
        Track jobs that spawn multiple nodes, e.g. HPC submission
        Include OSG distributed unprivileged Singularity to the search path and do a full test of Singularity
        Improved HTCondor configuration for schedd in Factory and Frontend
        Added flag to Ignore entries in downtime when considering glideins matches
        Add a bind mount for /etc/hosts in Singularity
        Removed Frontend service dependency from HTCondor service
        Bug fix: Factory GlideinMonitor* classads appear to be erased periodically
        Bug fix: Glidein not killing condor processes
        Bug fix: Error preventing the Frontend to match jobs
        Bug fix: Make sure metasite limits are respected
        Bug fix: Convert per/frontend limits to integers before division
        Bug fix: 'tochild' AttributeError in Factory
        Bug fix: Pilot proxies could be generated with invalid VOMS ACs
        NOTE: To use the new HTCondor configuration you need to reconfig or restart HTCondor after merging eventual .rpmnew files

v3_4_3  Add a scaling factor for all glideins limits in the entries
        Comprehensive unit-test for the parsing of the Frontend configuration with collectors using shared port
        Add the possibility to disable completely Glidein removal
        Increase verbosity to help Singularity troubleshooting
        Improve error messages in glideFactoryCredentials
        Automatically remove glideins after walltime
        Bug fix: Improve manual_submit_glidein
        Bug fix: Improve the way condor_jdl dict is populated for metasites
        Bug fix: Factory crashing with malformed HTCondor log (AttributeError: dirSummaryTimingsOut.data)
        Bug fix: glidein_config parsing cutting the results at the first space
        Bug fix: shell scripts improvements (condition smart_partionable.sh, variable splitting)
        Bug fix: Frontend upgrade is failing if it is unable to determine the version of the Factory
        Bug fix: Avoid glideFactoryEntryGroup processe leaks
        Bug fix: Stale running and held glidein numbers reported in factory classads
        Bug fix: Update documentation about ports requirement for Frontend's submit host
        Bug fix: The factory seems to ignore the configuration values in the files in the config.d directory w/ entry configurations
        Bug fix: Frontend not recognizing entries in downtime
        Bug fix: Pylint tests improved (htcondor setup, remove external lines)
        NOTE: If you update from 3.4 or earlier make sure to have open port 9618 also on standalone Frontends/subit hosts (was using 9615 before)

v3_4_2  Controlling that Frontend is not using options incompatible w/ linked Factories
        Use systemctl for loading/unloading on EL7

v3_4_1  Improved Singularity support (improved negotiation, added bind path variables and GPU support)
        Switch User collector child collectors to shared_port
        Updated documentation: remove obsolete parts, verified all references.
        Increased unit test coverage
        Added the possibility to skip idle removal per entry
        Improved how subentries are picked for metasites
        Bug fix: Periodic scripts fails but report GLIDEIN_PS_OK=True
        Bug fix: Fix entry_set configuration not to load temporary files and to work across upgrades
        Big fix: fixed metasites monitoring

v3_4    GlideinWMS 3.4 includes all features of previous stable series 3.2 and development series 3.3
        Inform the admin in case multiple service reload is done in rapid succession
        Released glideinwms-switchdoard RPM to provide condor_root_switchboard
        Singularity wrapper using PATH and module when SINGULARITY_BIN does not contain the correct path
        Update of the software license
        Have frontend groups glidein requests track the job requests
        Add a resource_slot that adds CPUs to the system but is in the main slot
        singularity jobs each use a separate linux session to support restricted-access CVMFS
        Do not set GLIDEIN_ToDie based on X509 user proxy expiration
        Estimate the cores provided to glideins running on an entry
        Write LogCompletedStats jsons for monitoring purposes
        Add entry monitoring breakdown for metasites
        Code modernization to Python 2.6/2.7 stansards (futurize stage 1 compliant)
        Increased testing (more than doubled unit tests coverage and testing more files w/ pylint)
        Bug Fix: TypeError: under special conditions in Factory reconfig
        Bug Fix: Entries in downtime slow down the ramp-up for the frontend
        NOTE: The type of the GLIDEIN_CPU attr is String (to accommodate the keywords auto, slot, node). Documentation was reporting Int incorrectly. Make sure your configuration uses the correct type or you may get a reconfig/upgrade error.
        NOTE: If you use HTCondor 8.7.2 or bigger with the GlideinWMS Factory, you must install also glideinwms-switchboard, to have condor_root_switchboard, that was removed from HTCondor
        NOTE: 'entry_sets' should be considered an experimental feature: the imlpementation is changing and there may be errors when upgrading across versions

v3_3_3  Includes all features and bug fixes released in v3_2_22_2

v3_3_2  Allow multiple remote directories for BOSCO submissions
        Bug fix: Submit attributes in entry configuration are now transmitted to AWS VM
        Documented vm_id_fname and vm_type_fname
        Includes all features and bug fixes released in v3_2_18
        Includes some important features and bug fixes planned for v3_2_19
          (fix of AWS submission)

v3_3_1  Includes all features and bug fixes released in v3_2_16

v3_3    Includes all features and bug fixes released in v3_2_15
        Support Google Compute Engine CE. Requires glidein cloud vm rpms v2+
        Support native configuration of EC2 spot prices and AZ in the entry
        Support frontend policies specified in external python file
        Support changes to VM ID and VM Type without need to reconfig/upgrade
          frontend service
        Bug Fix: Multiple credential (eg: vm_id+vm_type) definitions now work
	  correctly
        Bug Fix: create_condor_tarball now also includes required globus libraries
	  from lib/condor that HTCondor loads at runtime

v3_2_22_2 Changed singularity options also in singularity_setup.sh to adapt to the new singularity release requirements
        Bug Fix: Fix bug due to malformed -order option (SOFTWARE-3163)

v3_2_22_1 Changed singularity options to adapt to the new singularity release requirements

v3_2_22 Bug Fix: Incorrect behavior of Singularity
        Bug Fix: proxy-renewal-script updates and bug fixes
        Bug Fix: Critical bug in 3.2.21 leads to leaking glideFactoryEntryGroup.py processes

v3_2_21 Have frontend groups request the removal of unused glideins
        Support of unprivileged singularity and new singularity scripts
        Automatically renew gwms proxies
        Factory monitoring displaying correctly core counters
        Balancing glidein pressure to sites that are aliases or Meta-Sites
        Remove osg-version requirements
        Bug Fix: Fix fork.py behavior
        Bug Fix: Uninitialized variable caused skipping fix-rrd
        Bug Fix: Sanitize content of MJF attributes
        Bug Fix: Update Google custom search to new API

v3_2_20 Verified that Factory can start 1500 entries at a time
        Improved Factory monitoring by adding cores count for running and idle jobs and requests
        Removed dependency from Globus clients
        Support Singularity (future replacement for glexec)
        Avoid race conditions with multiple reload quick invocations in SL7
        Bug Fix: -fix_rrd is always in the upgrade command beside reconfig option
        Bug Fix: Fixed some failing unit tests
        Bug Fix: Fixed Factory job stats are empty
        Bug Fix: Bad link to Frontend monitoring
        Bug Fix: Custom Google search in the documentation pointing to the new site,
          not the old mirror on uscms.org
        Bug Fix: Fix DC_DAEMON_LIST
        Bug Fix: verifyRRD hard coded paths, missing some files during -fix_rr
        Bug Fix: HTCondor QEdit triggered also when advertise_pilot_accounting is not set
        Bug Fix: Downtime setting correctly distinguish daylight saving
        Bug Fix: Added usercollector RPM dependency from ganglia

v3_2_19 Added counters for Idle jobs older than X hours
        Enabling GWMS_XSLT_PLUGIN_DIR by default for
          glideinwms-vofrontend-standalone rpm
        Linked Frontend monitoring from Factory monitoring
        Improved glideins scale down by adding a timeout, GLIDEIN_IDLE_LIFETIME
        Log number of activation/claims per glidein
        Several documentation improvements, including reconfig and upgrade
          operations in SL7 RPM installation
        Bug Fix: reduced the number of file descriptors used per Entry in the Factory
        Bug Fix: Factory entries submitting glideins even after hitting the limit
        Bug Fix: Errors with HTCondor 8.5/6 upgrade: SUBSYS.LOCALNAME.* warning
          triggered by GWMS htcondor configuration, Glidein sent unintentionally
          to multiple schedds on the Factory
        Bug Fix: Clarified attribute types and fixed globbing behavior
        Bug Fix: RPM verification fails when config files are changed
        Bug Fix: GLIDEIN_CPU settings "node" and "slot" supported also in Frontend,
          better documented and improved to compensate for PBS misconfiguration
        Bug Fix: Fixed submission to AWS which uses key_pair as auth_method

v3_2_18 Bug Fix: ProjectId is missing double quotes
        Bug Fix: Stdout messages from startup script in SL7 are confusing
        Bug Fix: Service definition file for SL7 is marked executable
          and it should not
        Bug Fix: Upgrade not working when frontend/factory are running
        Bug Fix: Use of daemon function to start process does not play well
          with non empty pid files

v3_2_17 Glideins fallback to curl if transferring files from the
	  factory/frontend staging area fails using wget
	Bug Fix: Factory not correctly consider the cluster size when
	  doing multiple submissions during same cycle
	Added support for systemctl init scripts for RHEL6 and RHEL7
	Factory and Frontend service init scripts now use daemon functions
	Added ability to request disk space for special resource slots
	Updated jQuery in monitoring webpages to version 1.12
	Updated Documentation
	Pilot accounting information from jobs is now available in the
	  glidein job's classad on the factory side
	Frontend service performance stats are now advertised in the
	  glidefrontendmonitor classad
	Bug Fix: Fixed race condition where periodic scripts in glidein
	  would corrupt a glidein_config
	Bug Fix: Fixed an issue where some START_EXPR were incorrectly
	  ignored
	Bug Fix: When not configured GLIDEIN_CPUS default back to 1
	  rather than trying to auto detect available cores
	Bug Fix: Glideins will not use UTMP be default

v3_2_16	Information about job and machine features are now available in glidein
	  startd's classad
	New tool manual_glidein_startup now lets you manually start a glidein
	  for a given factory entry and frontend group
	BOSCO entries can now accept credentials provided from the frontend
	Bug Fix: Glidein is now correctly auto detect RHEL6 and RHEL7
	Bug Fix: Dagman and Schedd universe jobs are not counted against
	  max jobs running
	Bug Fix: An entry in downtime will now show up in the glidein status
	Bug Fix: Added condor-python rpm as a dependency
	Bug Fix: Complex credentials (vm_id+vm_type) are now correctly
	  interpreted
	Bug Fix: Frontend is now more resilient to transient errors
	  communicating with the HTCondor daemons
	Bug Fix: Slave frontend now correctly looks up the master frontend
	  when using condor-python bindings
	Bug Fix: create_condor_tarball now correctly includes required globus
	  libraries from lib/condor that are loaded by HTCondor at runtime
	Bug Fix: Frontend now correctly calculates the ReqMaxGlideins for
	  single core glidein entries
	Bug Fix: Numerical data in glideclient classad is not quoted to
	  preserve the data type
	Bug Fix: Frontend will not request any glideins at entry that is
          in downtime

v3_2_15	Factory will now automatically remove unrecoverable glidein jobs
	  with forcex if they are held for 20 times or more
	Several X509 related attributes from the matching job are now
	  also available in the glidein's startd's classad
	Prefix for attributes created by the periodic scripts is now
	  customizable
	If the glidein detects that a worker node is marked for draining and
	  if it is approaching the drain time, it will now kill the user job
	Frontend can be configured to request idle glideins at all times
	  irrespective of the jobs in the queue
	Bug Fix: Fixed a bug in the frontend downtime command in the script
	   /etc/init.d/gwms-frontend
	Bug Fix: Frontend now correctly considers group credentials before
	  frontend's global credential
	Bug Fix: Installing and upgrading GlideinWMS rpms now correctly
	  trigger httpd and htcondor reload commands in case of EL7
	Bug Fix: Fixed a bug where frontend would crash with pickling error
	  while using htcondor-python bindings
	Bug Fix: Factory now correctly remembers pilots submitted using
	  RFC proxy
	Bug Fix: Fixed a bug where running reconfig or upgrade command on
	  a frontend service would crash

v3_2_14_1 Added support for python in EL7
	Updated glidein_startup.sh to make it compatible with factory changes
	  coming in v3_2_15
	Bug Fix: Updated condor_config files used by the Schedd to work with
	  different versions of HTCondor including 8.4.7

v3_2_14	Various curbs and limits triggered in the factory are now logged
          in the glidefactory and glidefactoryclient classads
	Added initial support for python that comes with EL7
	Monitoring stats from factory completed logs are now advertised in
	  the glideresource classads
	Glideins can now shutdown themselves if the worker node is marked for
	  draining with appropriate messages logged in the glideins output
	Condor classad fetching is now done by using python bindings by
	  default. Until now this was done using condor_q and condor_status
	  commands
	Various limits configured in the factory and frontend are now
	  advertised in the respective classads
	Updated documentation
	Bug Fix: Factory will not release glideins sent to Condor CE if they
	  are held with authentication/authorization issues
	Bug Fix: Factory will not release glideins sent to AWS in case of
	  certain types of HoldReason
	Bug Fix: Proxies used by the glidein now if delegated have their
	  lifetime as long as the original proxy
	Bug Fix: Fixed issue where RRD processing was incorrectly throwing
	  ImportError instead of NameError
	Bug Fix: Frontend policies now correctly work if classad attributes
	  like RequestCpus are classad expressions
	Bug Fix: Fixed an issue where a slave frontend in HA mode would
	  crash if the WMS collector was down

v3_2_13	Support XSEDE ProjectId as a credential in frontend
	Glidein jobs can now auto detect cpus based on the sites WMS
	Frontend configuration settings idle_glideins_per_entry,
	  running_glideins_per_entry, running_glideins_total and
	  running_glideins_total_global now consider slots (startd classads)
	  reported in the User collector
        gwms-logcat.sh tool can now forward logs to a folder or http/https url
	Bug Fix: Factory will not release any glideins is max_per_cycle
	  in release section of config is set to 0
	Bug Fix: Frontend now accepts an attr with type="expr" as a condor
	  expression
	Bug Fix: Fixed several issues in the accounting of multi core
	  glideins
	Bug FIx: Counting of idle, running and total jobs in case of multi
	  core glideins is done correctly
	Bug Fix: Frontend will now correctly request enough glideins at
	  sites that support multi core glideins
	Bug Fix: Frontend group limits are now correctly applied based on
	  on the slots rather than glidein (condorg) jobs
	Bug Fix: Fix accounting bug where number of running cores would
	  log a negative count in case of multi core glideins
	Bug Fix: Fixed several issues with the cron type scripts

v3_2_12_1 Bug Fix: Fixed incompatibility with python 2.4 and bad failure when
          there is no entry, both introduced in v3_2_12 factory configuration

v3_2_12 Various curbs and limits triggered in the frontend are now logged
          in the glideresource classads
        Frontend is now more conservative while computing max request
          running
        Glideins now support advertising custom resources on the worker
          node This can be used to advertise resources like GPUs. GPUs can also
          be auto discovered and advertised
        Several improvements to rpm packaging. Useful frontend tools are
          now available in the user path
        Support of version control for the factory configuration and splitting
          of entries configuration from the main factory configuration (entries.d)
        Unique idle jobs matched by the frontend is now available in
          glideresource classads
          deloyment specific configuration and entry specific configuration.
        Bug Fix: Fixed a bug where CCB_ADDRESS configuration for the
          glidein was not created correctly under certain conditions
        Bug Fix: create_frontend script now correctly populates images
          in the monitoring pages
        Bug Fix: gwms-logcat now correctly supports multiple users
        Bug Fix: Frontend now correctly deadvertises glideresource
          classads on shutdown
        Bug Fix: Disable collector's use of shared port to support
          HTCondor 8.4 (in both factory and user pool)
        Bug Fix: Counting correctly glidein and cores, specially for
          partitionable slots
        Bug Fix: Fixed bug where DaemonShutdown was failing to consider
          dynamic slots
        Bug Fix: Fixed bug where NUM_CPUS was not set for partitionable slots
          resulting in hardware cpu number being used instead of GLIDEIN_CPUS

v3_2_11_2 Bug Fix: Fixed authentication issue introduced in v3_2_11 where a
          glidein startd fails to send keep alive signals to v8.2.x schedds

v3_2_11_1 Bug Fix: Fixed a bug introduced in v3_2_11 where file period
          interpreted as number instead of string

v3_2_11 VO Frontend now blacklists schedds with CurbMatchmaking=True
        You can now over provision Multicore glidein by using
          GLIDEIN_Resource_Slots attribute to specify different types of
          resources it provides. For example ioslot
        Glidein can now advertise itself to the site's local HTCondorCE
          collector if CONDORCE_COLLECTOR_HOST is set in it's environment
        Custom/validation scripts can now be run periodically and not
          just at the glidein's startup
        Improvements to the rpm packaging
        Updated documentation
        Bug Fix: Glideins in claimed/idle status are not shutdown by
          DAEMON_SHUTDOWN expression
        Bug Fix: Fixed a bug in gwms-logcat tool
        Bug Fix: The CCB selection behaves correctly and accepts sinful
          strings
        Bug Fix: Fixed a bug where Frontend under certain conditions
          would crash because of uninitialized ha_mode
        Bug Fix: Setting GLIDEIN_Report_Failed to ALIVEONLY now creates
          valid invalidate command
        Bug Fix: Fixed a bug in the accounting of jobs run by a glidein

v3_2_10 Improved Documentation
        Bug Fix: Fixed several bugs in accounting of idle and running slots
          in case of multicore glideins
        Bug Fix: Got rid of old style HTCondor default Memory & Disk
          requirements in the schedd configuration
        Bug Fix: DAEMON_SHUTDOWN expression will now let the Multi core glidein
          to run for appropriate time before killing it

v3_2_9  VO Frontend supports a master-slave HA mode
        Added a factory wrapper script to view glidein logs files
        Updated the dependency of GlideinWMS to HTCondor v8.2.2
        Frontend supports CCBs in addition to User Collector
        Updated documentation
        Bug Fix: glideresource classads now contain appropriate monitoring
          information
        Bug Fix: Fixed a bug where an unhandled exception would cause a
          frontend to shutdown
        Bug Fix: Removed absolute default requirement for vanilla jobs in
          user schedd's config file
        Bug Fix: Glidein now works correctly when both grid/voms-proxy
          commands are not available on the worker node
        Bug Fix: GlideinWMSVersion is now correctly reported in rpm distribution

v3_2_8  VO Frontend parameters are added to HTCondor config for ganglia
          monitoring
        CONDOR_VIEW_HOST is now set to localhost for factory collectors to
          minimize overhead in communication between primary and secondary
          collectors
        Added option to compress process logs in factory and frontend
        Added failed glidein statistics to frontend monitoring
        Added idle/running/total core statistics to frontend monitoring
        Added the support for HTCondor GANGLIAD monitoring
          (requires HTCondor 8.1 or newer). If you have HTCondor 8.0.x or earlier
          you must remove <t>/etc/condor/config.d/01_gwms_ganglia.config</t>,
          otherwise your HTCondor will complain about an unsupported option
          and crash.
        USE_CCB is now enabled by default and this information is
          advertised in the glidefactory classads
        Improved documentation
        Bug Fix: Glideins do not mail admins when HTCondor daemon crash
        Bug Fix: Gridmanager log paths used by glidein/factory scheds are
          now correctly expanded for different users
        Bug Fix: Factory and Frontend service start/stop exit codes now
          confer to Linux standards
        Bug Fix: Fixed issue where work dir and versioning in frontend config
          would break the config in case of frontend rpms
        Bug Fix: Made improvements to the HTCondor configuration used by
          factory rpm
        Bug Fix: There is no name collision for glideins when
          USE_PID_NAMESPACES is enabled in site's HTCondor batch system
        Bug Fix: Factory does not leak file descriptors when HTCondor
          commands using privilege separation fail

v3_2_7_2 Bug Fix: Set MASTER.USE_SHARED_PORT instead of USE_SHARED_PORT to
          avoid secondary collectors using the shared port daemon

v3_2_7_1 Set USE_SHARED_PORT to get around the issue with HTCondor 8.2.3

v3_2_7  Glideins now have an option to report monitoring info to a
          different collector configured in the factory
        Glideins now support shared port
        Glideins now use local storage for its tmp internal operaations
        Improved documentation
        Bug Fix: Fixed an issue where a corrupted internal state file
          would crash the factory
        Bug Fix: KeyError in a match_expr is now correctly logged
        Bug Fix: proxy_url in an entry's config is now correctly used
        Bug Fix: rrdtool commands are now used correctly when rrdtool
          python library is not installed on the system
        Bug Fix: Error classads now correctly advertise all the relevant
          attributes
        Bug Fix: glidein_off now correctly work with the HTCondor HA setup
        Bug Fix: Internal security changes are now properly cleaned up
          and applied in factory and frontend
        Bug Fix: Factory monitoring now correctly report UserRunning info
          when frontend is configured with multiple credentials
        Bug Fix: Fixed an issue with the factory rpm installation in case
          of privilege separation
        Bug Fix: Secondary schedd for frontend is now disabled by default
        Bug Fix: DAEMON_SHUTDOWN in glidein now uses idle timers that
          are relative to change in the state
        Bug Fix: Factory rpm now properly pulls down dependencies
        Bug Fix: UpdateSequenceNumber for classads now update correctly
        Bug Fix: Frontend now correctly provisions multicore glideins if
          the GLIDEIN_CPUS is configured for the entry
        Bug Fix: GLIDEIN_MaxMemMBs_Estimate now takes GLIDEIN_CPUS in the
          consideration

v3_2_6  condor_chirp is now added to condor tarbar used by glidein
        Added support for submitting glideins to batch sites using BOSCO.
          Requires HTCondor v8.2.2+
        Added new tool to purge old glideins
        Added periodic auto-update to Status Now monitoring pages
        Upon completion, glidein Job history is brought back to the factory
        Allow for separation of Factory collector and CondorG collector
        Bug Fix: Fixed local timezone in some frontend monitoring pages
        Bug Fix: Improved frontend performance
        Bug Fix: Requesting single-core partitionable glideins is not allowed
        Bug Fix: Fixed file ownership issues for rpm packages

v3_2_5_1 Bug Fix: Fixed an issue with the factory_startup template that affects factory reconfig/upgrade in case of RPM

v3_2_5  Added administrative commands for frontend fetch_glidein_log,
          glidein_off and enter_frontend_env
        Frontend now considers MAX_JOBS_RUNNING when requesting more glideins
        Frontend can now perform several tasks in parallel making it
          more scalable
        Frontend and Factory startup scripts are more consistent with each
          other
        Improved Documentation
        Bug Fix: Fixed an issue when factory config with HTCondorCE attributes
          would result in an invalid XML on reconfig
        Bug Fix: Fixed a bug where number of jobs run as reported by a
          glidein was significantly scaled up
        Bug Fix: Fixed issues in frontend introduced in v3_2_4 where the
          frontend would crash under certain conditions
        Bug Fix: Frontend's group logging and factory logging now
          correctly consider the backup_count when configured
        Bug Fix: Frontend RPM now correctly creates frontend.xml config with
          default trust_domain='grid' for the credentials
        Bug Fix: Fixed frontend performance issue introduced in v3.2.4

v3_2_4  Added support for HTCondor-CE attributes in the factory
        Made several performance improvements to frontend. Frontend does
          several tasks in parallel to better utilize the CPU.
        Factory & frontend monitoring pages now use new javascriptrrd v1.1.0+
        Factory monitoring now aggregates Log RRDs
        Frontend can now limit total idle glideins
        Added limits to globally total idle glideins
        Added badput summary line in the factory report
        Improved documentation
        Bug Fix: Factory & frontend operations like reconfig and upgrade now
          check if they are run by valid users
        Bug Fix: Fixed partitioning of multi-core glideins
        Bug Fix: Fixed bug in factory/frontend stopping
        Bug Fix: Fixed several bugs in the /etc/init.d/gwms-factory script
          available through the RPM distribution
        Bug Fix: Fixed bug with the factory/frontend monitoring that resulted
          in significantly scaled up monitoring numbers when frontend used
          multiple proxies
        Bug Fix: Factory now properly advertises entries in downtime
        Bug Fix: GLIDEIN_Glexec_Use when defined in the group now correctly
          overrides the value defined in global scope

v3_2_3  Glideins now have the ability to track the worker node batch slot.
          Based on the batch system at site (HTCondor, SGE, PBS, LSF, SLURM) the
          information is reported in the glidein's STARTD classad and logged in
          job's log file written by HTCondor using classad variables
          GLIDEIN_SiteWMS, GLIDEIN_SiteWMS_JOBID, GLIDEIN_SiteWMS_QUEUE and
          GLIDEIN_SiteWMS_SLOT.
        Number of rotated process_logs for factory and frontend process to
           keep can now be configured using backup_count configuration
           attribute
        Bug Fix: Factory now correctly updates the renewed credentials it gets
          from the Frontends
        Bug Fix: Cloud related configuration attributes VM_DISABLE_SHUTDOWN and
          VM_MAX_LIFETIME are now documented
        Bug Fix: Partitionable slots now correctly evaluate daemon shutdown
        Bug Fix: Partitionable slots now correctly coalesce when the jobs finish
        Bug Fix: For non-rpm installs, reconfiguring the factory from outside
          the factory working directory now works correctly
        Bug Fix: Frontend with no credentials configured logs appropriate info
          in the log files
        Bug Fix: clone_glidein tool is now packaged with the factory rpm
        Bug Fix: Factory now correctly cleans up completed_jobs logs
        Bug Fix: Improved Frontend efficiency by reducing the calls to OpenSSL

v3_2_2  Bug Fix: Fix a bug where factory would crash if it fails to query
          client global classads in wms collector
        Bug Fix: Glidein jobs correctly interpret the ARC CE FINISHED state

v3_2_1  Added support for a plug-in architecture for config that lets admins
          manipulate frontend and factory config with ease.
        Bug Fix: Factory accounting now correctly accounts for held jobs.
        Bug Fix: Improved error reporting in case of misconfigured
          credentials
        Bug Fix: Improved error reporting when factory fails to startup
        Bug Fix: Improved factory performance during log cleanup.
        Bug Fix: Fixed a bug introduced in v3_2 where glidein update
          interval was too short that resulted in performance issues for
          busy collectors.

v3_2    Added support for generation of condor tarball from condor installed
          via rpm
        Bug Fix: Fixed a race condition while shutting down factory service
        Bug Fix: Fixed a bug where glidein would not set certain condor config
          variables correctly
        Bug Fix: Fixed a bug where analyze_queues and analyze_entries would not
          work with the http:// URI
        Bug Fix: Collector port ranges in frontend.xml are now validated for
          errors
        Bug Fix: Schedd name, frontend name and identity in factory config
          are validated for errors
        Bug Fix: Starting a factory with all entries disabled now prints
          helpful message
        Bug Fix: Glidein does not leak LD_LIBRARY_PATH to job's environment
        Bug Fix: Added several speed enhancements to the factory to get
          looptime under acceptable limits
        Bug Fix: Fixed a bug where factory would not query rpm installed
          schedd correctly
        Bug Fix: Fixed a bug where factory would throw exception while logging
          during aggregating monitoring information
          frontend
        Bug Fix: Fixed a bug in glidein where it would not report back to
          all the user pool collectors in HA mode correctly
        Bug Fix: Fixed a bug where some of the parameters passed to glideins
          were not escaped correctly
        Bug Fix: Submitting glideins in test only mode now works correctly
        Bug Fix: Fixed a bug where factory would not submit glideins when
          privilege separation is disabled
        Bug Fix: Fixed a bug in manageFactoryDowntimes.py where it could not
          find required python libraries
        Bug Fix: Fixed a bug in factory accounting where glideins in certain
          state were not accounted towards idle state
        Bug Fix: Added support for the ACCEPTED state for ARC CE
        Bug Fix: factoryStatus.html now correctly auto-selects timezone
        Bug Fix: Fixed a bug where RSL for NorduGrid CE was not populated
        Bug Fix: glidefactory classads now correctly display GlideinWMSVersion

v3_1    Includes relevant features and bug fixes released upto v2_7
        Added a new feature that enables glidein to advertise error classads
          to the User Collector to help in debugging
        Added a new tool add_entry to help with entry creation
        Support frontend to auto-update and auto-generate proxies using
          proxy creation tools
        Added config conversion tool for factory and frontend to convert v2
          based config to new format
        Frontend can periodically execute external proxy renewal/creation
          scripts to keep the frontend proxies updated
        Glideinwms now uses standard python logging APIs
        Version 3 Factory is backward compatible with the Version 2 Frontends
        Provide tools convert_frontend_2to3.sh and convert_factory_2to3.sh to
          convert the version 2 based config files to version 3 format
	Glidein Monitoring slots are now disabled by default
	Factory process now tries to increase RLIMIT_NPROC if possible
        Improved documentation
        Bug Fix: DAEMON_SHUTDOWN now works with the multi-slot glideins
        Bug Fix: Tools analyze_queues and analyze_frontend now correctly work
	  with the http:// URI
        Bug Fix: Fixed a bug where glidein would not parse some of its
	  configuration variables correctly
	Bug Fix: Fixed a bug where create_glidein would fail when used with
	  condor 7.9.4+
        Bug Fix: Factory does better error reporting when an entry is configured
          with an invalid sched_name
        Bug Fix: Fixed bug where log files were not rotated correctly

v3_0_0  Change to max job limits to reflect more accurate names
	Add attr_dict in the environment of the frontend match_expr
	New feature: The start_expr is now a native part of match
	Support for new API for factory - frontend communication
	Improved logging using standard Python logging APIs
        The option to have either the factory and/or the frontend provide the
	  pilot proxy has been removed.  The frontend must always provide it.
	Added support to the Factory for EC2 Query API submissions to the Cloud
	Consolidated the condor_tarball tag in glideinwms.xml to read in a list
	  of os, arch, version
        Added unittests for downtime, proxy plugins, and support functions
        Miscellaenous bug fixes

v2_7_2  Bug Fix: Fixed a race condition while shutting down factory service
        Bug Fix: Fixed a bug where glidein would not set certain condor config
          variables correctly
        Bug Fix: Fixed a bug where analyze_queues and analyze_entries would not
          work with the http:// URI
        Bug Fix: Collector port ranges in frontend.xml are now validated for
          errors
        Bug Fix: Schedd name, frontend name and identity in factory config
          are validated for errors
        Bug Fix: Starting a factory with all entries disabled now prints
          helpful message
        Bug Fix: Glidein does not leak LD_LIBRARY_PATH to job's environment
        Bug Fix: Fixed a bug where stopFactory would send two TERM signals
          too quickly to stop processes

v2_7_1  glidecondor_adddn tool now supports adding dns from a file
        added new tool glidecondor_createseccol to dynamically create
          secondary collectors
        bug fix: fixed a bug where condor config templates used by glideinwms
          components had undefined values
        bug fix: analyze_entries tool now correctly handles uri http://
        bug Fix: Fixed a bug where frontend would crash if the some of the
          internal files were truncated to zero length.
        Bug Fix: Factory now correctly does the cleanup of log files
        Bug Fix: Factory now correctly does accounting of glideins in default
          condor schedd
        Bug Fix: proxy_info works correctly with the cat option

v2_7    glideinwms code is now organized as python packages
        Factory no longer creates long running process per entry. Instead, it
          runs only a single long running glideFactoryEntryGroup process
          instead of multiple glideFactoryEntry processes. Factory config
          supports two additional config attributes entry_parallel_workers
          and update_thread_count (see factory configuration docs for details)
        Improved factory prerformance by migrating popen calls to use python's
          subprocess library
        Improved the performance of the Frontend by using pre-clustering in
          the Frontend match-making
        Created index.html for factory monitoring so that admins can disable
          directory browsing in httpd.conf
        Added support for Partitionable condor slots in the glidein
        Frontend policy for matching factory classads is now exposed in
          glideresource classads
        Added a tool to dynamically create a secondary schedd config files
          and directories to the existing installation
        Improved the usability of the frontendGrouGraphStatusNow monitoring
        Factory now ships with two new operations tools: entry_ls and entry_rm
        Improved documentation
        Factory will now try to recover glideins with held code 79
        Bug Fix: Fixed a bug where installers were not setting
          SHARED_PORT_MAX_WORKERS in respective condor_config correctly
        Bug Fix: Fixed a bug where the frontend would match jobs with now
          proxy to sites that require glexec
        Bug Fix: Fixed a bug where the factory entry processes and the frontend
          group (element) processes would inherit their parent's environment
          corrupting their own environment
        Bug Fix: Glidein's job wrapper now behaves as true wrapper
        Bug Fix: Fixed a bug where frontend_startup would not correctly pick
          default frontend.xml
        Bug Fix: Fixed a bug in frontend where failure to evaluate match_expr
          for one group would prevent frontend from requesting glideins for
          other groups
        Bug Fix: RPM now has shared port daemon enabled in the configuration
        Bug Fix: Fixed a bug where Factory would sometime use wrong DN when
          calculating proxy hash
        Bug Fix: Fixed a bug where factoryStatusNow.html would not link to the
          factoryEntryStatusNow.html correctly if the entry name is too long
        Bug Fix: setup_x509.sh now correctly checks for existence of the
          grid-proxy-info and voms-proxy-info binaries on the worker node
        Bug Fix: Frontend now counts the partitionable slots correctly

v2_6_2  Factory now supports glidein failure modes. This information is
          propagated back to the factory.
        Glideins now have different idle timeouts for startup and tail. The
          glideins now exit much faster after comleting jobs and they do not
          have enough time left to run another job.
        Added support for javascriptRRD 2.6.2
        Consolidate the condor_tarball tag in glideinwms.xml to read in a list
          of os, arch, version. See docs for details.
        Ini-installer will default collector_port to 9618 if not specified in
          appropriate config sections.
        Installation now supports GRIDMANAGER_PROXY_REFRESH_TIME for condorg
          services
        Improved logging when frontend fails to evaluate match expressions.
        Monitoring pages changed to be even more compliant with DOM4
        Improved the means for distributing condor config files for different
          condor services.
        Improved documentation.
        Bug Fix: Better handling of monitor dir during upgrade
        Bug Fix: Factory correctlt handles frontend in downtime and now it does
          not affect requests from other frontends
        Bug Fix: Putting entry in downtime multiple times now warns the user
        Bug Fix: Fixed a bug where v2.6.1 frontend would crash under certain
        Bug Fix: Installers do not set CONDOR_VERSION as constant by default.
          This will now enable frontends to override this attr.
        Bug Fix: Ini-installer now correctly cleans up files on re-install

v2_6_1  Factory now limits the speed at which it releases held glideins. After
          too many unrecoverable attempts, held glideins are removed.
        Added support for Condor 7.8+
          KNOWN ISSUE: Only applies if you upgrade condor binaries in place
          without using the installer. Upgrading condor binaries for user pool
          and the wms collector with multiple schedds should not be done without
          corresponding changes to the condor configuration file. If the
          JOB_QUEUE_LOG is not correctly configured for each sched, condor
          queue will be corrupted for all the scheds
        Glideins now support curl & data file transfer plugins in condor
        New installations of factory and frontend services now communicate with
          condor daemons using TCP by default.
        Bug Fix: Factory configuration now supports specifying limits for
          different security classes of same frontend
          KNOWN ISSUE: If limits are configured in the factory, this change
          breaks backward compatibility. As a workaround, admin needs to remove
          the limits and re-apply them.
        Bug Fix: Factory monitoring pages now correctly work in Firefox 14+
        Bug Fix: Frontend now monitors the glideins correctly when the
          corresponding factory classad is missing
        Bug Fix: Fixed factory logging. Factory can now correctly extract info
          from a voms proxy
        Bug Fix: Factory now correctly logs the Completed jobs info in rrd

v2_6    Add attr_dict in the environment of the frontend match_expr
        Config defined attributes are now available in match_expr
        Factory now supports deleting entries using --force_delete
        Upgrading glideinwms with changes to the monitoring rrds is supported
        Frontend advertises its monitoring url to the factory
        Misbehaving glidein can be put on hold using WANT_HOLD
        The start_expr is now a native part of match
        Startd now advertises per-slot memory when configured by the factory.
          VO frontend can use GLIDEIN_MaxMemMBs_Estimate attr to make glidein
          estimate the available memory (based on memory/core or memory/cpu)
        entry_q now reads GLIDEIN_FACTORY_DIR from env before checking cwd
        Multiple user collectors are supported in HA mode
        Factory can specify if glexec required vs provided by site
        Providing config file locations for glideinwms.xml and frontend.xml is
          optional during reconfig
        Entry downtime reason is now reported in the glidefactory classad
        Improved support for RHEL 6 platform
        Added support for new ARC Gatekeeper 1.x
        Linux distro of the worker node is available in the glidein's classad
        Condor classad fields are now considered case insensitive
        Bug Fix: Java is correctly disable by default preventing increase in
          the image size during condor daemon benchmarking
        Bug Fix: Glidein correctly handles semicolons in STARTER_JOB_ENVIRONMENT
        Bug Fix: Frontend correctly correctly supports <attr> properties
        Bug Fix: Glidein now correctly finds glexec on glite site
        Bug Fix: Glideins update the user collector using TCP by default
        Bug Fix: Ini-Installer now correctly works with condor tarball on RHEL6
        Bug Fix: Reduced the logging in factory generated by inactive entries
        Bug Fix: For frontend rpm, init.d scriptes now correctly sudo to the
          frontend user before reconfig
        Bug Fix: Any errors due to changes in condor_q output are logged
        Bug Fix: MaxJobRetirementTime and PREEMPT expressions now correctly
          work with WANT_HOLD
        Bug Fix: Factory correctly deals with the situation when the frontend
          changes the proxy used
        Bug Fix: It is possible to specify limits for different frontends for
          an entry in the factory config
        Bug Fix: Glidein sets the LD_LIBRARY_PATH so condor uses local shared
          libraries before those in system path
        KNOWN ISSUE: When the monitor work_dir is moved, reconfig/upgrade will
          fail. Recommended work-around is to delete the xml configuration in
          the work directory, change the work directory, then reconfig/upgrade.
        KNOWN ISSUE: If the work directory has changed, frontend startup
          upgrade will not create the group directories and then the frontend
          will fail (silently). If you are changing directories (e.g. from a RPM
          upgrade), you should copy the group directories manually.


v2_5_7
	Patch of the clean_glidein_queue to return 1 only when something was
	  really done
	Fix handling of held jobs when hitting the held limit
	Fix log messages that printed out wrong held limits
	Add locking when talking to the collector

v2_5_6  Factory supports per-frontend limits in the config
        Bug Fix: Fixed a bug where frontend would not stop cleanly
	Bug Fix: Factory does not depend on the X509_CERT_DIR in the environment

v2_5_5	Added filtering to analyze_entries
	Reduced the number of debug log messages when entry becomes inactive.
	Frontend match making uses autoclusters, greatly increasing the
	  performance for busy frontends
	Package the list of shared libraries required by Condor 7.7.3+ for
	  glidein tarballs
	Pilot proxy is not available in the user job environment any more
	  reducing the chance for user job using it.
	Ini Installer can now create a template for single service installs
	Bug Fix: Fixed several issues with the frontend rpm
	Bug Fix: GLIDEIN_Glexec_Use=NEVER setting now correctly works with the
	 setting require_voms_proxy=True
	Bug Fix: Installer now correctly works with the RPM OSG client
        Bug Fix: ReqMaxIdle was grossly overestimated before.
        Bug Fix: Fixed bug in factory monitoring web pages where the link
          generation was off by one for monitoring groups
        Bug Fix: Correctly respect the remove max_per_cycle=0 glideins in the
          factory
        Bug Fix: glexec_setup.sh now uses correct proxy for its tests

v2_5_4  Added support for VDT pre installed via rpm. While using
          ini-installer set vdt_location to empty and
          x509_cert_dir=/etc/grid-security/certificates
        Consolidated some of the frontend rpm related patches
        Added support for rsl in glideinWMS.xml to map to cream_attributes
          in the jdf
        Bug Fix: Factory now correctly renews the proxies if the limits are hit
        Bug Fix: frontend_startup does not show "upgrade" in help
        Bug Fix: glideclient, glidefactoryclient & glideresource classads now
          correctly show UpdateSequenceNumber to help condor track potentially
          lost updates
        Bug Fix: Fixed a bug where condor_shared_port daemon was not correctly
          stopped under certain conditions

v2_5_3  Updated the license to reflect current Fermitools license
        Factory does a better job at respecting frontend limits
        Admins can add custom html to the monitoring pages
        Monitoring now displays additional information about the Factory and
          Frontend names
        Improved documentation
        Enable Match Authentication configuration by default
        Made querying of schedds efficient where ever possible.
        Add autocomplete/search to table in frontendGroupGraphStatusNow.html
        Added 'upgrade' option to factory startup script so that all the
          files in the glidein submit directory are updated.  Did the same for
          the frontend startup script. 'reconfig' only updates the files
          populated with information from the configuration files (no scripts).
        The OSG rpm worker node client requires changes to glidein startup
          scripts. If you plan on using OSG sites that have used this rpm, you
          must "upgrade" your factory in order to get glideins.
        Bug fix: If there is no GLOBUS_LOCATION available to glidein, it will
          not exit
        Bug fix: Fixed a bug where factory would not advertise the entry under
          certain conditions when it was put in downtime.
        Bug fix: Fixed a bug where glidein_reconfig would not properly validate
          schedd_name
        Bug fix: Fixed a bug where frontend reconfig was not populating
          GSI_DAEMON_PROXY correctly in frontend.condor_config
        Bug fix: Writeback of config files on reconfig is now enable by default
        Bug fix: Frontend does better error reporting when querying user pool or
          the schedd fails
        Bug fix: Fixed a bug in condor log parsing when we encounter event 400
          (Job was evicted)
        Bug fix: Fixed a bug where factory would not correctly accept requests
          from frontends that were still using old keys
        Bug fix: Fixed a bug where glidein pilot proxy lifetime was not taken
          into account vs GLIDEIN_Max_Walltime
        Bug fix: Fixed a bug where expired/renewed proxy created error in
          completed_jobs accounting
        Bug fix: Fixed a bug where running_glideins_per_entry was not treating
          the limits correctly
        Bug fix: Fixed a bug in DAEMON_SHUTDOWN
        Bug fix: Fixed a bug where voms requirements were not treated correctly
        Bug fix: Added glidecondor_upgrade back to the install dir. This file
          was missing in v2_5_2_1 release

v2_5_2_1Bug fix: Fixed a bug creation of START expression when VOMS proxy is
          set be required.
        Bug Fix: Revert the glidein shutdown behavior introduced in v2_5_2
        Bug Fix: Add condor_glexec_update_proxy to Condor tarball if available

v2_5_2  Frontend publishes glideresource classad to the pool collector. This
          is useful for the users to do match making for their jobs.
        Useful job info like JOB_Site, JOB_GLIDEIN_Entry_Name, JOB_GLIDEIN_Name
          JOB_GLIDEIN_Factory, JOB_GLIDEIN_Schedd, JOB_GLIDEIN_ClusterId,
          JOB_GLIDEIN_ProcId, JOB_GLIDEIN_Site is now added to SUBMIT_EXPRS
        Default GLEXEC_JOB = True when GLEXEC_BIN is set to use glexec
        Factory can now reuse old public key after startup to decrypt
          frontend requests. Grace time for old public key is configurable.
        The frontend now can set global limits on number of glideins.
        The frontend now can use more than a single CPU.
        glexec now tested during initial validation.
        Bug fix frontend: Do not double count glideins when using multiple
          groups.
        Bug fix frontend: Respect per-entry running limits.
        Bug fix factory: The factory was not properly checking the security
          classes and setting downtimes.
        Bug fix factory: Improper termination of glideins because of
          SIGHUP is handled correctly
        Bug fix factory: Daylight savings is now correctly accounted for.
        Bug fix factory: GLIDEIN_Max_Walltime is now used correctly.
        Major improvements in the factory and frontend monitoring.
          Now requires javascriptRRD 0.6.0+.
        Added tools for comparing the Factory configuration file with what
          is published in information systems.
        Limit the max number of glideins per frontend
        Use DAEMON_SHUTDOWN to shutdown glidein daemons
        Allow factory to specify if an entry point (CE) requires voms proxies
          only for pilot and user jobs
        Documentation added:
          - Secondary WMS/User Collector documentation added to Advanced
          Condor Configuration
          - Documented the internal communication protocol through classads

v2_5_1  Factory now can remove excessive glideins
          New ClassAd attribute - RemoveExcess
        Frontend will request removal of glideins when no user jobs in queue
        Improved idle counting by the frontend when jobs match many factory
          entries.
        Improved frontend logging.
        Add Java support in the glideins.
        Changed factory monitoring. It is now based on security names and not
          plain frontend names
        Improved installation process using ini based installer.
        GlideinWMS tarball does not include CVS directories and files any more
        Added support of shared port configuration for condor daemons
        SIGHUP signals to glideins are ignored correctly.
        Improved documentation
        BUG FIX: Fixed a bug introduced in v2_5 release where reconfiguring the
          factory would fail because of monitoring groups
        BUG FIX: Fixed a typo in one of the links in frontendRRDBrowse.html

v2_5	Installer can now install gridFTP and VOMS certs needed by CREAM
	GlideinWMS release is now available in 3 different tarballs, frontend
	  only, factory only and a complete tarball.
	GlideinWMS factory and the Corral frontend can now talk to each other
	Factory is smarter about handling held glideins
	Factory can now black/white list VOs on a per-entry basis
	Version of GlideinWMS is now published in the classads. Scheme can
	  detect any changes to any service appropriate files and advertises
	  cersion as patched.
	Frontend should try to recover the crashed group before it gives up and
	  shutsdown
	Improvements to the monitoring
        Improvements to documentation
	Add support for use of TCP in condor_advertise
	Add support for condor_advertise -multiple (requires Condor v7.5.4+)
	Improved factory stopping procedure
	Add XML monitoring files of RRD files on both factory and frontend
	Add JobsRunningHere attribute - CANNOT SIMPLY UPGRADE, NEEDS NEW
	  INSTANCE FOR BOTH FACTORY AND FRONTEND
	Graceful shutdown of the glidein by trapping signals in glidein_startup
	BUG FIX: Fixed a bug where factory would create a malformed
	  glideinWMS.xml config file when configured to use a default proxy
	  for glideins from the factory.
	BUG FIX: Factory entry sometimes stops reporting when it gets an
	  exception for any reason
	BUG FIX: Top-level schedd_status.xml malformed Total data
	BUG FIX: Fixed a bug where the LogCounts.rrd was created with wrong
	  data types.
	KNOWN ISSUE: Automatic release of held glideins for CREAM CEs with
	  issues could result in factory submitting more than required
	  glideins to the CE. As a workaround, disable release of held glideins
	  for CREAM CE.

v2_4_3  Install VOMS Certificates during the install time
	Entry does not print stacktrace when it fails to submit glidein.
	  Instead it logs the error message appropriately.
	Default scripts run on the worker node correctly print errors to
	  stderr instead of stdout
	Allow factory to startup from any directory
	Python scripts get the python from /usr/bin/env python instead of
	  /bin/env python making them more portable across different linux
	  distros
	Better exception handling and error reporting
	Fixed how voms_proxy_init is looked for after VDT install
	BUG FIX: Fixed a bug where Summarize.listStored() in CondorMonitor.py
	  returned the wrong value
	BUG FIX: Fixed a bug in factory monitoring that prevented proper
	  aggregation of sites
	BUG FIX: Correctly interpret DN from a voms proxy generated from a
	  service certificate.
	BUG FIX: Generate frontend's condor_config without empty values for
	  certain configuration options. Empty values is not the right way to
	  reset the configuration options.


v2_4_2	Fixed the incompatibility introduced in v2_4_1 in monitoring components.

v2_4_1	Add a new configuration option for PREEMPT_GRACE_TIME
	Move configuration of GLIDEIN_Job_Max_Time from the factory to the
	  frontend
	In factory config added checks to make sure Factory configuration is
	  consistent w.r.t CONDOR_ARCH, CONDOR_VERSION, CONDOR_OS
	Monitoring enhancement to make selection/deselection of groups/entries
	  easier
	Support GLIDEIN_Glexec_Use in frontend config attrs. Frontend can
	  mandate or make the use of GLEXEC optional. Used in conjunction with
	  GLEXEC_BIN in factory config. If GLEXEC_BIN in set to NONE for an
	  entry in factory config, it is assumed that the entry doesn't have
	  GLEXEC configured on site.
	Improvements to documentation
	BUG FIX: Allow the use of the same DN for both the security and the
	  collector
	BUG FIX: Returning invalid variable during proxy creation
	KNOWN ISSUES: Running GlideinWMS v2_4_1 with Condor 7.5.3 has not been
	  tested. Although, most of the things may work, security changes in
	  Condor may affect your installation.


v2_4	Add proxy security classes to the frontend
	Add SecurityName to the frontend.
	Add DNs for frontend proxy and all the daemons the frontend talks to;
	  create own Condor config file and gldiein gridmap file out of them.
	Frontend now dynalically creates GLIDEIN_Collector.
	Frontend now requires Match authentication.
	Add frontend authentication info to the factory config.
	Add support for different frontend identities in different WMS
	  collectors.
	Put log files in a separate tree.
	Put client logs and client proxies in separate trees.
	Implement privilege separation in the factory.
	Various minor refactoring of code to achieve the above.
	Aggregate gatekeeper/sites for factory monitoring.
	Introduced monitoringgroups to group sites together for monitoring
	  purposes in factory.
	KNOWN ISSUES: Installer is not always able to get the correct
	  DN from the certificates/proxies. It does not correctly strip the
	  CN=<blah> bit of the DN in certain cases. If the DN guessed is not
	  correct, make the required changes in the configurations and/or
	  condor_mapfile.

v2_3_2	Fixed a security bug in the way certificate/proxy DN(s) are handled. If
	  you are upgrading the installation rather than full install, you
	  should patch the condor_mapfile used by Condor daemons in your
	  installation using the convert_condormap tool available from the
	  GlideinWMS download page. For additional security, users should
	  add new DN to the condor_mapfile only by using the glidecondor_addDN
	  tool available in the GlideinWMS/install directory.

v2_3_1	Updated documentation. This is document change only release
	Introduced monitoringgroups to group sites together for monitoring
	  purposes in factory.
	KNOWN ISSUES: Installer is not always able to get the correct
	  DN from the certificates/proxies. It does not correctly strip the
	  CN=<blah> bit of the DN in certain cases. If the DN guessed is not
	  correct, make the required changes in the configurations and/or
	  condor_mapfile.

v2_3	Attempt to restart a crashed entry few times before shutting down the
	  factory. Restarting is allowed for max restart_attempts in time
	  interval restart_interval sec in the factory.
	Add vacuum option to manageDowntimes.
	Factory now properly handles new-style frontends without a group.
	Frontend code has been refactored to allow use as a library.
	More protections in place.
        Fixed a security bug in key handling. BREAKS BACKWARDS COMPATIBILITY! But it is needed.
	Fixed a bug in glidein_startup.sh that prevented the passing of * as a parameter value
	Fixed a logical bug in the glidein config that resulted in job preemption.
	Added support for condor_ssh (v7.4 and up)
	Add support for unquoted string to be published in classads. The type is 'expr'
	Documentation Changes
	Minor bugs fixed.
 	KNOWN ISSUES: If using condor 7.4.0-7.4.2 and 7.5.0, USE_VOMS_ATTRIBUTESshould be set to false for Collector and negotiator to avoid potential problems and memory leaks in GSI libraries.

v2_2	Bug Fix: GLEXEC_JOB and GLEXEC_STARTER were not published in glideins
          classads. This was preventing pseudo interactive monitoring to work
          in case of GLEXEC.
	Made VDT optional.
	Added support for VDT 2.0, and made it the default.
	  Added Globus-Client and Myproxy-Client in the minimal VDT install.
	Installer now allows the collector to run on non-standard port.
	Factory now only checks X509_USER_PROXY if it needs it.
	Added support for multiple versions of condor in a single factory.
	Added ReqEncIdentity to the frontend->factory protocol
	   to prevent replay attacks.
	   WARNING: This effectively prevents old-style frontends to talk
                    to new style factories.
	   NOTE: Will only work with Condor 7.3.1 or newer
	KNOWN ISSUES: Pseudo interactive monitoring will not work with glexec.

v2_1	Fixed a bug that was throwing an exception if a glidein failed.
	Fixed handling of grid-mapfiles coming from client.
	Fixed support for nordugrid.
	Improvements to the monitoring.
	Improvements to the installer.
        Improved documentation.

v2_0	Refactoring creation libs;
	  create_frontend and create_glidein now share much code
	Added create_frontend, recreate_frontend.
	Frontend now has a stage web area and passes it to the factory
	  Factory publishes list of supported signtypes and
            frontend uses this for factory selection
	  User provided code in the glidein API changed
	    The 2nd arg is now one of main|entry|client|client_group
	Added tools/glidein_status.py.
	The monitoring page now has client-side monitoring based on
            javascriptRRD.
	Support multiple proxies... frontend drives this via plugins.

v1_6_3 Fixed a security bug in way certificate/proxy DN(s) are handled. If
          you are upgrading the installation rather than full install, you
          should patch the condor_mapfile used by Condor daemons in your
          installation using the convert_condormap tool available from the
          GlideinWMS download page. For additional security, users should
          add new DN to the condor_mapfile only by using the glidecondor_addDN
          tool available in the glideinWMS/install directory.

v1_6_2	Fixed a bug where Frontend would crash during certain conditions.

v1_6_1	Better randomization of GCBs and collectors
	Collector list now supports ranges of ports
	Fixed Condor-G log parsing
	Better treatment of multiple-collectors in the installer
	Add support for CCB and USE_MATCH_AUTH in the installer.
	Add OSG:vo-client to the minimal VDT install
	Better handling of Nordugrid sites.
	Start factory in nice mode to give priority to Condor daemons
	In condor_config for glideins remove explicit STARTER_UPDATE_INTERVAL
	Removing GLIDEIN_Use_TCP, now uses UPDATE_COLLECTOR_WITH_TCP instead
	Force integrity on reads from WMS collector
 	Better layout of monitoring pages
	Better error handling in Factory downtime management
	Better formatting of error logs for factory and frontend
	Bug Fix: GLEXEC_JOB and GLEXEC_STARTER were not published in glideins
	  classads. This was preventing pseudo interactive monitoring to work
	  in case of GLEXEC
	Improved documentation

v1_6	The XML file now supports comments.
	Installer uses both .profile and .bashrc.
	Frontend now requires integrity checks when talking to
	  the WMS collector.
	Refactored install options so that most tasks performed
          as a non-privileged user.
	Added create_frontend, recreate_frontend.
	Refactoring creation libs;
	  create_frontend and create_glidein now share much code
	Using m2crypto for sha1 checks.
	Added support for CCB.
        Monitoring refactored
          Most rrd operations now in separate module
          Grouped multiple attributes in the RRD files (lower overhead)
          Removed RRD locking
	Fixed Condor-G log parsing.
	Fixed startup bugs in various debug tools.
	Improved monitoring scalability (fewer RRD files)

v1_5_2	Fixed VDT installation
	Factory now changes public key at each restart.
	Minor monitoring improvements.

v1_5_1	Added flag that allows/requires proxies from frontend.
	Fixed bug in factory installation with encryption.
	The glidein now finds out about OSG squid.
	Fixed bug in the node blacklisting code.

v1_5	Added support for secure info passing between
          VO frontend and gfactory
          Requires M2Crypto Python module
        VO frontend can now give the gfactory int own proxy
          to be used for pilot submission
          Requires the encryption mentioned above
        gcb_setup is now loaded by default
          Can use GCB_ORDER=NONE to disable it
          Warning: Explicit listing of gcb_setup may create problems.
        glexec_setup is now loaded by default
	  Can use GLEXEC_BIN=NONE to disable it
          Warning: Explicit listing of glexec_setup may create problems.
	Randomized the retire time to smooth terminations.
	  Can be controlled via GLIDEIN_Retire_Time_Spread.
	Some monitoring tweaks.
	  Added more switches; xml incompatible with v1_4+
	The factory config file is now read only.
	Added switches to limit log file growth.

v1_4_4	Add factory_constraint parameter to the frontend.
	Add support for GLEXEC_JOB.
	Add support for USE_MATCH_AUTH.
	Add new protection against condor_submit errors.
	Warnings go into the info file, too. Easier to correlate errors this way.
	Better colors in graphs.

v1_4_3	Improve scalability
 	  by adding locks into monitoring
	  Now only one rdd update can proceed at a time, and
	  only one graphing operation can proceed at a time

v1_4_2	Fix glexec setup script bug.
	Improve pseudo-interactive monitoring.
	  The changes introduced in v1_4_1 could leave behind zombies.

v1_4_1	Improved pseudo-interactive monitoring
	  By default use a separate startd for monitoring
	  Old multi-VM mode can be enabled by MONITOR_MODE=MULTI
	Fixed factory handling held jobs
	Improved installer defaults

v1_4	Better support for RESS and BDII
	 Including automatic downtime handling
	User STARTD_SENDS_ALIVES=True by default.
	Disable glidein UDP port by default.
	Global files can now be loaded after entry
	Added support for job wrapper scripts.
	Fix monitor locking.
	gcb_setup now supports ORDER=NONE.
	gcb_setup is now loaded automatically.
	Add verbosity to entries.
	Add config section to entries.
	Improve held handling.
	Add a completed job log.
	Improve signal handling (for stopping factory)
	Made the most CPU intensive part of monitoring optional.
	Installer now supports multiple collectors.

v1_3	Factory and frontend now gets sleep and advertise attr from config.
	Add downtime concept to the factory and relative management tool.
	Add command to get info about factory config file.
	Add init.d style startup script.
	Fix pseudo-interactive monitoring when glexec is used.
	Reduce VO frontend condor_q load by adding job_attributes.
	VO frontend now queries the collector and advertises result.
	Smarter algorithm to calculate min_idle, also using condor_status.
	Improve monitoring.

v1_2_3	Fix glexec handling in condor v7.0.3
	Cut in half the number of condor_q's in the frontend.
        Minor monitoring twaeks.
        Minor installation tweaks.

v1_2_2	Fix condor installation.
        Update versions in installer.
        Added glidein_gcb and collector_setup.
	Add LIBEXEC to condor_config.
        Better support for EGEE.

v1_2_1	Added support for BDII.
        Added stop scripts for factory and frontend.
        Added tools/wmsTxtList.py and tools/glidein_interactive.py.
        All commands now are executable.
        Add client and Condor-G monitoring in factory data.
        Greatly improved log stats monitoring.
        Add doc section on monitoring.

v1_2	More fine grained configuration of glideins
        Factory reads now force integrity checks
        Glideins now publish the gatekeeper name.

v1_1	Entry points can now be updated by using reconfig_glidein
        Add automatic OSG glexec discovery
        Use condor_mapfile for authorization
        Installer now supports Condor v7 (without Quill)
        Installer now supports gLExec
        FIXED SECURITY CONFIGURATION, REINSTALL EVERYTHING

v1_0	During installation, Quill is now optional.
        During installation, Condor config can be split into condor_config.local.
        Files are now loaded in order specified.
        Added local_start.sh to ease testing.
        Slimmed down glidein_submit.sh.
        Many bugs fixed.
        NOT BACKWARD COMPATIBLE: Entry points need to be recreated.

v0_9	Significant speedup in the VO frontend matchmaking
	Added MaxRunningGlideins request
        Use human readable dates in logs
        Factory will use python rrd module if present
        Many bugs fixed.
        NOT BACKWARD COMPATIBLE: Entry points need to be recreated.

v0_9	Significant speedup in the VO frontend matchmaking
	Added MaxRunningGlideins request
        Use human readable dates in logs
        Factory will use python rrd module if present
        Reduced graphing load of the factory
        Frontend will work even if some of the schedd are down
        Improved installation scripts.
        NOT BACKWARD COMPATIBLE: Need to recreate both frontend and factory entries

v0_8	Add support for multiple schedds
	Add initial support for pseudo interactive monitoring
	Improved Web monitoring
	Better documentation
	NOT BACKWARD COMPATIBLE: Entry points need to be recreated.

v0_7	Add support for Condor 6.9.2 => subdirs in condor tarball
	Add support for user variables
	Move log and monitoring files into entry dir
        Add lock files
        Create proper ClassAd cleanup when daemons exit.
        NOT BACKWARD COMPATIBLE: Entry points need to be recreated.

v0_6    The glidein factory now serves multiple entry points.
        The config file is now XML based.
        NOT BACKWARD COMPATIBLE: Entry points need to be recreated.

v0_5    More bug fixes.
        Added monitoring info in the classads.
        Improved monitoring info of the Factory.

v0_4	Many bug fixes.
	Added RRD graphs to glidein factory.

v0_3	Added support for GCB and gLExec
	Better stability and logging.
	Many bug fixes.
	Greatly improved user documentation.
	NOT BACKWARD COMPATIBLE: Entry points need to be recreated.

v0_2	First fully configured version
	Very few things are still hardcoded here

v0_1    The first fully functional version
        Needs more polishing and better configuration, but it works
