# GlideinWMS The Glidein-based Workflow Management System

- Home
- Download
- Glidein Frontend
- WMS Factory
- Components
- FAQ

## Components

- Overview
- Prerequisites
- User Pool
- User Schedd
- GSI
- HTCondor

### HTCondor Configuration Attributes

Jump to:

Related Links:

- HTCondor Website
- More GlideinWMS HTCondor Topics

## Reference Guide to HTCondor Configuration Attributes

The tables below acts as a reference guide for condor_config attributes for GlideinWMS.

GlideinWMS Service | Attribute | Description
ALL | CONDOR_IDS = 500.500 | HTCondor user (uid.gid)
CONDOR_ADMIN = user at domain | Contact, via email, when problems occur
UID_DOMAIN=$(FULL_HOSTNAME) |
FILESYSTEM_DOMAIN=$(FULL_HOSTNAME) | With glideins, there is nothing shared
LOCK = $(LOG) | HTCondor lock files to synchronize access to various log files. They use the log directory so they are collocated.
DAEMON_LIST = MASTER, COLLECTOR, NEGOTIATOR, SCHEDD | HTCondor daemons. The WMS Collector services will have the list shown above. The User Collector service uses all but the SCHEDD. The Submit service uses only the SCHEDD. The Frontend service is a client only instance and will have a 'DAEMON_SHUTDOWN = True' to insure no daemons are started accidently.
SEC_DAEMON_SESSION_DURATION = 50000 | Limit session caching to 12 hours
SEC_DEFAULT_AUTHENTICATION = REQUIRED SEC_DEFAULT_AUTHENTICATION_METHODS = FS,GSI SEC_READ_AUTHENTICATION = OPTIONAL SEC_CLIENT_AUTHENTICATION = OPTIONAL DENY_WRITE = anonymous@* DENY_ADMINISTRATOR = anonymous@* DENY_DAEMON = anonymous@* DENY_NEGOTIATOR = anonymous@* DENY_CLIENT = anonymous@* | Authentication settings
SEC_DEFAULT_ENCRYPTION = OPTIONAL SEC_DEFAULT_INTEGRITY = REQUIRED SEC_READ_INTEGRITY = OPTIONAL SEC_CLIENT_INTEGRITY = OPTIONAL SEC_READ_ENCRYPTION = OPTIONAL SEC_CLIENT_ENCRYPTION = OPTIONAL | Privacy settings
GSI_DAEMON_TRUSTED_CA_DIR = full_path_to_CA_directory | Certificate directory
GSI_DAEMON_CERT = full_path_to_your_hostcert.pem GSI_DAEMON_KEY = full_path_to_the_hostkey.pem | Credentials
GSI_DAEMON_DIRECTORY = full_path_to_your_condor_mapfile_directory | Set daemon cert location
CERTIFICATE_MAPFILE = full_path_to_the_condor_mapfile | HTCondor mapfile
HOSTALLOW_WRITE = * ALLOW_WRITE = $(HOSTALLOW_WRITE) | With strong security, do not use IP based controls
GSI_DAEMON_NAME= | Identity of the GlideinWMS services used to authenenticate when communicating between serrvices. This is a comma separated list of their identities (DNs).
WMS Collector User Collector | NEGOTIATOR_POST_JOB_RANK = MY.LastHeardFrom | Prefer newer claims as they are more likely to be alive
NEGOTIATOR_INTERVAL = 60 NEGOTIATOR_MAX_TIME_PER_SUBMITTER=60 NEGOTIATOR_MAX_TIME_PER_PIESPIN=20 | Increase negotiation frequency, as new glideins do not trigger a reschedule
PREEMPTION_REQUIREMENTS = False | Prevent preemption
NEGOTIATOR_INFORM_STARTD = False | Negotiator/GCB optimization
NEGOTIATOR.USE_VOMS_ATTRIBUTES = False | Disable checking for VOMS extended attributes (FQAN)
NEGOTIATOR_CONSIDER_PREEMPTION = False | Causes Negotiator to run faster. PREEMPTION_REQUIREMENTS and all condor_startd rank expressions must be False for NEGOTIATOR_CONSIDER_PREEMPTION to be False
COLLECTOR_NAME = wms_service |
COLLECTOR_HOST = $(CONDOR_HOST):9618 |
COLLECTOR.USE_VOMS_ATTRIBUTES = False | Disable checking for VOMS extended attributes (FQAN)
##EVENT_LOG=$(LOG)/EventLog ##EVENT_LOG_JOB_AD_INFORMATION_ATTRS=Owner,CurrentHosts,x509userproxysubject, AccountingGroup,GlobalJobId,QDate,JobStartDate,JobCurrentStartDate, JobFinishedHookDone,MATCH_EXP_JOBGLIDEIN_Site,RemoteHost ##EVENT_LOG_MAX_SIZE = 100000000 | Event logging, if desired
##COLLECTOR_MAX_FILE_DESCRIPTORS = 20000 | Allow more file descriptors (only works if HTCondor is started as root)
COLLECTOR0 = $(COLLECTOR) COLLECTOR0_ENVIRONMENT = "_CONDOR_COLLECTOR_LOG=$(LOG)/Collector0Log" COLLECTOR0_ARGS = -f -p 9641 DAEMON_LIST = $(DAEMON_LIST), COLLECTOR0 | For each secondary HTCondor collector desired, the attributes specified must be set. For more detailed information on this, refer to the Advanced HTCondor Configuration - Mulitple Collectors document
WMS Collector Submit | MAX_JOBS_RUNNING = 6000 | Allow up to 6K concurrent running jobs.
JOB_START_DELAY = 2 JOB_START_COUNT = 50 | Start max of 50 jobs every 2 seconds
JOB_STOP_DELAY = 1 JOB_STOP_COUNT = 30 | Stop 30 jobs every seconds This is needed to prevent glexec overload, when used.
MAX_CONCURRENT_UPLOADS = 100 MAX_CONCURRENT_DOWNLOADS = 100 | Raise file transfer limits. No upload limits, since JOB_START_DELAY limits that. But do limit downloads, as they are asyncronous
APPEND_REQ_VANILLA = (Memory>=1) && (Disk>=1) JOB_DEFAULT_REQUESTMEMORY=1 JOB_DEFAULT_REQUESTDISK=1 | Prevent checking on ImageSize With 7.8.+, the APPEND_REQ_VANILLA is replaced by the JOB_DEFAULT_REQUESTMEMORY/DISK attributes. Having all 3 attributes defined does not present a problem.
MAXJOBRETIREMENTTIME = $(HOUR) * 24 * 7 | Prevent preemption
SCHEDD_SEND_VACATE_VIA_TCP = True STARTD_SENDS_ALIVES = True | GCB optimization
ENABLE_USERLOG_FSYNC = False | Reduce disk IO - paranoid fsyncs are usully not needed
SHADOW.GLEXEC_STARTER = True SHADOW.GLEXEC = /bin/false | Prepare the Shadow for use with glexec-enabled glideins
MAX_SHADOW_LOG = 1000000000 | Limits size of shadow logs
SEC_ENABLE_MATCH_PASSWORD_AUTHENTICATION = TRUE | Submit service only and only if you have enabled the USE_MATCH_AUTH attribute in the Factory and Frontend configuration files.
SHADOW_WORKLIFE = 0 | This is a workaround to a HTCondor problem if SEC_ENABLE_MATCH_PASSWORD_AUTHENTICATION = TRUE and HTCondor version <= 7.5.3 See Condor ticket 1481
SHADOW.USE_SHARED_PORT = True SCHEDD.USE_SHARED_PORT = True SHARED_PORT_MAX_WORKERS = 1000 SCHEDD.SHARED_PORT_ARGS = -p 9615 DAEMON_LIST = $(DAEMON_LIST), SHARED_PORT | SCHEDD Shared Ports
WMS Collector | SCHEDDGLIDEINS2 = $(SCHEDD) SCHEDDGLIDEINS2_ARGS = -local-name scheddglideins2 SCHEDDGLIDEINS2.SCHEDD_NAME = schedd_glideins2 SCHEDDGLIDEINS2.SCHEDD_LOG = $(LOG)/SchedLog.$(SCHEDDGLIDEINS2.SCHEDD_NAME) SCHEDDGLIDEINS2.LOCAL_DIR_ALT = $(LOCAL_DIR)/$(SCHEDDGLIDEINS2.SCHEDD_NAME) SCHEDDGLIDEINS2.EXECUTE = $(SCHEDDGLIDEINS2.LOCAL_DIR_ALT)/execute SCHEDDGLIDEINS2.LOCK = $(SCHEDDGLIDEINS2.LOCAL_DIR_ALT)/lock SCHEDDGLIDEINS2.PROCD_ADDRESS = $(SCHEDDGLIDEINS2.LOCAL_DIR_ALT)/procd_pipe SCHEDDGLIDEINS2.SPOOL = $(SCHEDDGLIDEINS2.LOCAL_DIR_ALT)/spool SCHEDDGLIDEINS2.JOB_QUEUE_LOG = $(SCHEDDGLIDEINS2.SPOOL)/job_queue.log SCHEDDGLIDEINS2.SCHEDD_ADDRESS_FILE = $(SCHEDDGLIDEINS2.SPOOL)/.schedd_address SCHEDDGLIDEINS2.SCHEDD_DAEMON_AD_FILE = $(SCHEDDGLIDEINS2.SPOOL)/.schedd_classad SCHEDDGLIDEINS2_ENVIRONMENT = "_CONDOR_GRIDMANAGER_LOG=$(LOG)/GridManagerLog.$(SCHEDDGLIDEINS2.SCHEDD_NAME).$(USERNAME)" DAEMON_LIST = $(DAEMON_LIST), SCHEDDGLIDEINS2 DC_DAEMON_LIST = + SCHEDDGLIDEINS2 | Atttributes for defining multiple schedds. When multiple schedds are used, each additional schedd must specify the unique location for the log file using JOB_QUEUE_LOG. The execute, lock and spool directories for each schedd must be created manually. For more detailed information on this, refer to the Advanced Condor Configuration - Mulitple Schedds document
SCHEDDGLIDEINS2_SPOOL_DIR_STRING = "$(SCHEDDGLIDEINS2.SPOOL)" SCHEDDGLIDEINS2.SCHEDD_EXPRS = SPOOL_DIR_STRING | Allows the Factory to query the schedds directly bypassing the collector and thus improving performance.
Submit | SCHEDDJOBS2 = $(SCHEDD) SCHEDDJOBS2_ARGS = -local-name scheddjobs2 SCHEDDJOBS2.SCHEDD_NAME = schedd_jobs2 SCHEDDJOBS2.SCHEDD_LOG = $(LOG)/SchedLog.$(SCHEDDJOBS2.SCHEDD_NAME) SCHEDDJOBS2.LOCAL_DIR_ALT = $(LOCAL_DIR)/$(SCHEDDJOBS2.SCHEDD_NAME) SCHEDDJOBS2.EXECUTE = $(SCHEDDJOBS2.LOCAL_DIR_ALT)/execute SCHEDDJOBS2.LOCK = $(SCHEDDJOBS2.LOCAL_DIR_ALT)/lock SCHEDDJOBS2.PROCD_ADDRESS = $(SCHEDDJOBS2.LOCAL_DIR_ALT)/procd_pipe SCHEDDJOBS2.SPOOL = $(SCHEDDJOBS2.LOCAL_DIR_ALT)/spool SCHEDDJOBS2.JOB_QUEUE_LOG = $(SCHEDDJOBS2.SPOOL)/job_queue.log SCHEDDJOBS2.SCHEDD_ADDRESS_FILE = $(SCHEDDJOBS2.SPOOL)/.schedd_address SCHEDDJOBS2.SCHEDD_DAEMON_AD_FILE = $(SCHEDDJOBS2.SPOOL)/.schedd_classad DAEMON_LIST = $(DAEMON_LIST), SCHEDDJOBS2 DC_DAEMON_LIST = + SCHEDDJOBS2 | Atttributes for defining multiple schedds. Effective with Condor 7.7.5+, a JOB_QUEUE_LOG attribute was added. When multiple schedds are used, each additional schedd must specify the unique location for that file. The execute, lock and spool directories for each schedd must be created manually. For more detailed information on this, refer to the Advanced Condor Configuration - Mulitple Schedds document
SCHEDDJOBS2_SPOOL_DIR_STRING = "$(SCHEDDJOBS2.SPOOL)" SCHEDDJOBS2.SCHEDD_EXPRS = SPOOL_DIR_STRING | Allows for queries against the schedds directly bypassing the collector and thus improving performance. Nothing has been implemented at this time to utilize this. Only Factory/WMS Collector communication uses this.
WMS Collector | GRIDMANAGER_MAX_SUBMITTED_JOBS_PER_RESOURCE=5000 GRIDMANAGER_MAX_PENDING_SUBMITS_PER_RESOURCE=5000 GRIDMANAGER_MAX_PENDING_REQUESTS=500 GRIDMANAGER_PROXY_REFRESH_TIME=604800 SCHEDD_ENVIRONMENT = "_CONDOR_GRIDMANAGER_LOG=$(LOG)/GridmanagerLog.$(USERNAME)" | Condor-G attributes The GRIDMANAGER_PROXY_REFRESH_TIME attribute forces Condor-G to re-delegate the proxy as soon as the Frontend provides one, defaulting to 1 week, since we do not expect proxies with longer lifetimes.
QUEUE_SUPER_USERS = $(QUEUE_SUPER_USERS), factory_user | Only required when privilege separation is in effect. Allows the Factory user to bypass privilege separation making administration simpler without compromising security.
Submit | JOB_Site = "$$(GLIDEIN_Site:Unknown)" JOB_GLIDEIN_Entry_Name = "$$(GLIDEIN_Entry_Name:Unknown) JOB_GLIDEIN_Name = "$$(GLIDEIN_Name:Unknown) JOB_GLIDEIN_Factory = "$$(GLIDEIN_Factory:Unknown) JOB_GLIDEIN_Schedd = "$$(GLIDEIN_Schedd:Unknown) JOB_GLIDEIN_ClusterId = "$$(GLIDEIN_ClusterId:Unknown) JOB_GLIDEIN_ProcId = "$$(GLIDEIN_ProcId:Unknown) JOB_GLIDEIN_Site = "$$(GLIDEIN_Site:Unknown) SUBMIT_EXPRS = $(SUBMIT_EXPRS) JOB_Site JOB_GLIDEIN_Entry_Name JOB_GLIDEIN_Name JOB_GLIDEIN_Factory JOB_GLIDEIN_Schedd JOB_GLIDEIN_Schedd JOB_GLIDEIN_ClusterId JOB_GLIDEIN_ProcId JOB_GLIDEIN_Site | User Job Class Ad Attributes