# GlideinWMS The Glidein-based Workflow Management System

- Home
- Download
- Glidein Frontend
- WMS Factory
- Components
- Recipes
- FAQ

## SciToken Authorization

- Overview
- Batch (BOSCO)
- Cloud Recipes
- Token Auth
- EC2 VM Roll

Jump to:

1. Description
2. Requirements
3. Obtain a SciToken
4. Configure Factory to submit your SciToken
5. Configure CE to accept your SciToken
6. Verify Token Functionality
7. Useful Links

## Description

This page documents a recipe for enabling and managing SciTokens Authorization between Glidein Factory Entries and the Compute Elements they submit to.

## Requirements

Requirement | Description
Compatible HTCondor and GlideinWMS Versions For both Factory and VOFrontend | Version numbers and installation instructions
An HTCondor Compute Element (htcondor-ce) Compatible with and correctly configured for SciTokens | Minimal version requirements at the time of this writing
    htcondor-ce.noarch          4.4.1-1.osg35.el7    @osg
    htcondor-ce-client.noarch   4.4.1-1.osg35.el7    @osg
    htcondor-ce-condor.noarch   4.4.1-1.osg35.el7    @osg
A SciToken that the htcondor-ce will accept | Example method for generating a working token is given below. This is not how SciTokens would be generated and used in a production environment.

## Obtain a SciToken

Method 1: follow instructions at https://opensciencegrid.org/technology/software/requesting-tokens/#requesting-tokens

Method 2: generate your own self-signed Scitokens. You need the python scitokens library and admin access to a web server that the Factory, Frontend, and CE can all access.

The following is an example of Method 2 on website 'jobsub.fnal.gov'

    1.  yum install python2-scitokens or pip install scitokens

    2. Generate some keys

    scitokens-admin-create-key --create-keys \
                               --pem-private > scitoken.private.pem

    scitokens-admin-create-key --private-keyfile scitoken.private.pem\
                               --jwks-private > test.scitoken.private.jwks

    scitokens-admin-create-key --private-keyfile test.scitoken.private.pem \
                               --jwks-public > test.scitoken.public.jwks

    3. copy keys over to jobsub.fnal.gov, create some other files in the document root directory:
    ssh fnalu.fnal.gov
    cd /web/sites/j/jobsub.fnal.gov/htdocs
    mkdir -p  oauth2/certs  .well-known
    # create file  .well-known/openid-configuration with these contents:
    cat .well-known/openid-configuration

    {
     "issuer":"https://jobsub.fnal.gov",
     "jwks_uri":"https://jobsub.fnal.gov/oauth2/certs/test.scitoken.public.jwks"
    }

    exit #return to original machine that you generated the keys on

    scp test.scitoken.public.jwks fnalu:/web/sites/j/jobsub.fnal.gov/htdocs/oauth2/certs

    3. Generate a token to perform condor operations on  an htcondor-ce. NOTE: the scitokens-admin-create-token utility installed by
       pip or yum generates tokens that are only valid for 10 minutes.  If this is not long enough for your tests copy the utility
       (it is a python script) and modify to taste.

    #for ce fermicloud349.fnal.gov
    scitokens-admin-create-token --key_id 7b50 --keyfile test.scitoken.private.pem \
                                 --issuer https://jobsub.fnal.gov sub="condor"\
                                 scope="condor"\
                                 aud="fermicloud349.fnal.gov:9619"  > fermicloud349.scitoken

    #for ce itb-ce2.chtc.wisc.edu
    scitokens-admin-create-token --key_id 7b50 --keyfile test.scitoken.private.pem\
                                 --issuer https://jobsub.fnal.gov sub="condor"\
                                 scope="condor"\
                                 aud="itb-ce2.chtc.wisc.edu:9619" > itb-ce2.chtc.wisc.edu.scitoken

## Configure Factory to submit your SciToken

1. Check that SCITOKENS authentication is enabled on the Factory. For example:

       [root@fermicloud359]# condor_config_val SEC_DEFAULT_AUTHENTICATION_METHODS
       SCITOKENS,TOKEN,FS,GSI

2. Place the correctly named, previously generated SciToken in the correct directory so that the VOFrontend knows to send it on to the Factory, which uses it to authenticate with the CE.

   In this example, the SciToken must be named el7_osg35.scitoken to associate it with Factory entry el7_osg35 The SciToken goes in directory
   /var/lib/gwms-frontend/tokens.d/

   As SciTokens expire, a mechanism must be in place to renew it periodically. A cron job running 'scitokens-admin-create-token' would suffice for development purposes, but some other mechanism would be used in a production environment.

## Configure CE to accept your SciToken

1. Check that htcondor on your CE accepts SCITOKENS as an authentication method:

       root@fermicloud349 ~]# condor_ce_config_val -dump | grep SCITOKENS
       COLLECTOR.SEC_ADVERTISE_STARTD_AUTHENTICATION_METHODS = FS,SCITOKENS,GSI
       COLLECTOR.SEC_READ_AUTHENTICATION_METHODS = FS,SCITOKENS,GSI
       COLLECTOR.SEC_WRITE_AUTHENTICATION_METHODS = FS,SCITOKENS,GSI
       MASTER.SEC_DEFAULT_AUTHENTICATION_METHODS = SCITOKENS, FS, GSI
       SCHEDD.SEC_READ_AUTHENTICATION_METHODS = FS,SCITOKENS,GSI
       SCHEDD.SEC_WRITE_AUTHENTICATION_METHODS = FS,SCITOKENS,GSI
       SCITOKENS_SERVER_AUDIENCE = $(COLLECTOR_HOST)
       SEC_CLIENT_AUTHENTICATION_METHODS = SCITOKENS, FS, GSI, PASSWORD
       SEC_DEFAULT_AUTHENTICATION_METHODS = SCITOKENS,FS, GSI

2. Check that an entry in the CE's CONDOR_MAPFILE assigns your SciToken to a valid user. In this example SciTokens issued by 'https://jobsub.fnal.gov' are mapped to user 'osg'

       [root@fermicloud349 ~]# condor_ce_config_val -dump | grep mapfile
       CERTIFICATE_MAPFILE = /etc/condor-ce/condor_mapfile
       [root@fermicloud349 ~]# cat /etc/condor-ce/condor_mapfile
       SCITOKENS https://jobsub.fnal.gov osg
       GSI (.*) GSS_ASSIST_GRIDMAP
       GSI "(/CN=[-.A-Za-z0-9/= ]+)" \1@unmapped.htcondor.org
       CLAIMTOBE .* anonymous@claimtobe
       FS "^(root|condor)$" \1@daemon.htcondor.org
       FS "(.*)" \1
       [root@fermicloud349 ~]#

## Verify Token Functionality

Here we see that the freshly installed SciToken can be decoded by command line tools and that condor_ping to the CE's collector authenticates some condor operations as user 'osg' as configured earlier in fermicloud349's condor_mapfile:

    [root@fermicloud359 ~]# _condor_SCITOKENS_FILE=/var/lib/gwms-frontend/tokens.d/el7_osg35.scitoken condor_ping -address "<$(host fermicloud349.fnal.gov | awk '{print $4}'):9619>" -table ALL
             Instruction Authentication Encryption Integrity Decision Identity
                   ALLOW      SCITOKENS       none       MD5    ALLOW osg@users.htcondor.org
                    READ           none       none      none    ALLOW unauthenticated@unmapped
                   WRITE           FAIL       FAIL      FAIL     FAIL FAIL  (use -verbose for more info)
              NEGOTIATOR           FAIL       FAIL      FAIL     FAIL FAIL  (use -verbose for more info)
           ADMINISTRATOR           FAIL       FAIL      FAIL     FAIL FAIL  (use -verbose for more info)
                   OWNER           FAIL       FAIL      FAIL     FAIL FAIL  (use -verbose for more info)
                  CONFIG           FAIL       FAIL      FAIL     FAIL FAIL  (use -verbose for more info)
                  DAEMON           FAIL       FAIL      FAIL     FAIL FAIL  (use -verbose for more info)
        ADVERTISE_STARTD      SCITOKENS       none       MD5    ALLOW osg@users.htcondor.org
        ADVERTISE_SCHEDD           FAIL       FAIL      FAIL     FAIL FAIL  (use -verbose for more info)
        ADVERTISE_MASTER           FAIL       FAIL      FAIL     FAIL FAIL  (use -verbose for more info)
    [root@fermicloud359 ~]#

Verify that submitting a job to the Factories entry point uses SciTokens authentication by adding D_SECURITY to the ALL_DEBUG condor setting on the appropriate condor daemons. The condor logs will show what authentication was used, and it should be SCITOKENS between the factory collector and the CE schedd.

A second verification method is to disable GSI communication between the Factory and the CE by commenting out the line in the Factories CONDOR_MAPFILE that does the GSI mapping for the CE, and submit a job. If the factory can still submit jobs to the CE, SCITOKENS authentication must be working.

## Useful Links

- SciTokens Claims and Scopes Language A good description of what claims you may want to put in your SciTokens and why.
- WLCG Common JWT Profiles WLCG Tokens are 'nearly' interoperable with SciTokens and are more mature.