# GlideinWMS The Glidein-based Workflow Management System

- Home
- Download
- Glidein Frontend
- WMS Factory
- Components
- Recipes
- FAQ

## WMS Factory

- Overview
- Details
- Configuration
- Design
- Monitoring
- Troubleshooting

### Tarball Downloader

Jump to:

1. Description
2. Exit Codes
3. Configuration
4. Options
5. Sample Execution
6. Sample Generated XML ile

Related Links:

- Factory configuration
- Frontend configuration

## Description

The get_tarball script is a Python script that downloads HTCondor tarballs from the official HTCondor website.
The tarballs are downloaded by looking at the URL set using TARBALL_BASE_URL in the configuration file. The files are downloaded and stored in the DESTINATION_DIR directory. Checkusm are verified after a file is downloaded. If a file already exist, or if its checksum is wrong or missing, the file is skipped.
A list of major condor version has to be specified using CONDOR_TARBALL_LIST. Each major version has to be a directory here (assuming this is the link set in the configuration). For each major version, all the tarballs in the WHITELIST are downloaded. Alternatively, all tarballs available are downloaded excluding the one in the BLACKLIST.
For each minor version (e.g.: "9.0.7"), the script attempts to download all the files in the FILENAME_LIST. The attempted URL is TARBALL_BASE_URL/MAJOR_VERSION/WHITELIST[i]/release/FILENAME_LIST[j], for example: https://research.cs.wisc.edu/htcondor/tarball/23.0/23.0.0/release/condor-23.0.0-x86_64_CentOS7-stripped.tar.gz
The next three parameters, OS_MAP, ARCH_MAP, and XML_OUT determines if and how an xml file with the tarballs has to be generated. If XML_OUT is not set then nothing is done and the other two parameters can be missing. Otherwise an xml file will be generated and stored there. OS_MAP and ARCH_MAP indicates how the operating system and the architecture in the tarball filename has to be translated in the xml corresponding fields.
get_tarball is a standalone script and does not require any glideinWMS dependency or installation.

## Exit Codes

- 0: All operations completed successfully.
- 1: Configuration file does not exist.
- 2: Cannot write XML_OUT file on disk.
- 3: Cannot find XML_OUT file when using --checklatest.
- 4: XML file needs to be updated since a newer version is present on the website.

## Configuration

To specify the configuration for the downloader, you can use the following methods:

- Set the GET_TARBALLS_CONFIG environment variable.
- Place a configuration file named get_tarballs.yaml in the same directory as get_tarball.py.

Sample Configuration File:

    DESTINATION_DIR: "/var/lib/gwms-factory/condor/"
    TARBALL_BASE_URL: "https://research.cs.wisc.edu/htcondor/tarball/"
    DEFAULT_TARBALL_VERSION: ["9.0.16"] # Can be set to "latest"
    CONDOR_TARBALL_LIST:
       - MAJOR_VERSION: "9.0"
         WHITELIST: ["9.0.7", "9.0.16", "latest"]
       - MAJOR_VERSION: "10.0"
         WHITELIST: ["latest"]
         CHECK_LATEST: True
       - MAJOR_VERSION: "10.x"
         DOWNLOAD_LATEST: True # Same as adding "latest" to a WHITELIST. Default False
       - MAJOR_VERSION: "23.0"
         WHITELIST: ["23.0.0"]
    FILENAME_LIST: ["condor-{version}-x86_64_CentOS7-stripped.tar.gz", "condor-{version}-x86_64_CentOS8-stripped.tar.gz",
        "condor-{version}-x86_64_AlmaLinux8-stripped.tar.gz", "condor-{version}-x86_64_Ubuntu18-stripped.tar.gz", "condor-{version}-x86_64_Ubuntu20-stripped.tar.gz",
        "condor-{version}-aarch64_Stream8-stripped.tar.gz", "condor-{version}-ppc64le_CentOS8-stripped.tar.gz", "condor-{version}-ppc64le_AlmaLinux8-stripped.tar.gz",
        "condor-{version}-aarch64_AlmaLinux8-stripped.tar.gz"]
    OS_MAP: { "CentOS7": "default, rhel7, linux-rhel7", "CentOS8": "rhel8, linux-rhel8", "AlmaLinux8": "rhel8, linux-rhel8", "Ubuntu18": "ubuntu18, linux-ubuntu18",
        "Ubuntu20": "ubuntu20, linux-ubuntu20" }
    ARCH_MAP: { "x86_64": "default", "ppc64le": "ppc64le", "aarch64": "aarch64" }
    XML_OUT: "/etc/gwms-factory/config.d/01-condor-tarballs.xml"

## Options

The script supports the following options:

- --verbose: Enables detailed output for debugging and tracking the script's operations.
- --checklatest: Ensures that the script verifies and downloads the latest available tarball for the specified versions.
  When this option is enabled, the script will actively check for updates in the tarball repository and ensure the most recent version of each specified tarball is downloaded.

## Sample Execution

This is the standard execution output. More output can be obtained using --verbose.

    $ get_tarballs
    Handling major version 9.0
    	File condor-9.0.7-x86_64_CentOS7-stripped.tar.gz successfully downloaded
    	File condor-9.0.7-x86_64_CentOS8-stripped.tar.gz successfully downloaded
    	File condor-9.0.7-x86_64_Ubuntu18-stripped.tar.gz successfully downloaded
    	File condor-9.0.7-x86_64_Ubuntu20-stripped.tar.gz successfully downloaded
    	File condor-9.0.16-x86_64_CentOS7-stripped.tar.gz successfully downloaded
    	File condor-9.0.16-x86_64_CentOS8-stripped.tar.gz successfully downloaded
    	File condor-9.0.16-x86_64_Ubuntu18-stripped.tar.gz successfully downloaded
    	File condor-9.0.16-x86_64_Ubuntu20-stripped.tar.gz successfully downloaded
    	File condor-9.0.18-x86_64_CentOS7-stripped.tar.gz successfully downloaded
    	File condor-9.0.18-x86_64_CentOS8-stripped.tar.gz successfully downloaded but
    	     checksum not available at https://research.cs.wisc.edu/htcondor/tarball/9.0/9.0.18/release/ (check 'sha256sum.txt')
    	File condor-9.0.18-x86_64_Ubuntu18-stripped.tar.gz successfully downloaded
    	File condor-9.0.18-x86_64_Ubuntu20-stripped.tar.gz successfully downloaded
    Handling major version 10.0
    	File condor-10.0.9-x86_64_CentOS7-stripped.tar.gz successfully downloaded
    	File condor-10.0.9-x86_64_AlmaLinux8-stripped.tar.gz successfully downloaded
    	File condor-10.0.9-x86_64_Ubuntu20-stripped.tar.gz successfully downloaded
    	File condor-10.0.9-ppc64le_AlmaLinux8-stripped.tar.gz successfully downloaded
    	File condor-10.0.9-aarch64_AlmaLinux8-stripped.tar.gz successfully downloaded
    Handling major version 10.x
    	File condor-10.9.0-x86_64_CentOS7-stripped.tar.gz successfully downloaded
    	File condor-10.9.0-x86_64_AlmaLinux8-stripped.tar.gz successfully downloaded
    	File condor-10.9.0-x86_64_Ubuntu20-stripped.tar.gz successfully downloaded
    	File condor-10.9.0-ppc64le_AlmaLinux8-stripped.tar.gz successfully downloaded
    	File condor-10.9.0-aarch64_AlmaLinux8-stripped.tar.gz successfully downloaded
    Handling major version 23.0
    	File condor-23.0.0-x86_64_CentOS7-stripped.tar.gz successfully downloaded
    	File condor-23.0.0-x86_64_AlmaLinux8-stripped.tar.gz successfully downloaded
    	File condor-23.0.0-x86_64_Ubuntu20-stripped.tar.gz successfully downloaded
    	File condor-23.0.0-ppc64le_AlmaLinux8-stripped.tar.gz successfully downloaded
    	File condor-23.0.0-aarch64_AlmaLinux8-stripped.tar.gz successfully downloaded

## Sample Generated XML File

An xml configuration file can be generated and placed in the /etc/gwms-factory/config.d/ directory to be picked up and used for the final glideinWMS.xml file. Here is a sample generated xml:

    <glidein>
       <condor_tarballs>
          <condor_tarball arch="default" os="default, rhel7, linux-rhel7" tar_file="/tmp/condor-9.0.7-x86_64_CentOS7-stripped.tar.gz" version="9.0.7"/>
          <condor_tarball arch="default" os="rhel8, linux-rhel8" tar_file="/tmp/condor-9.0.7-x86_64_CentOS8-stripped.tar.gz" version="9.0.7"/>
          <condor_tarball arch="default" os="ubuntu18, linux-ubuntu18" tar_file="/tmp/condor-9.0.7-x86_64_Ubuntu18-stripped.tar.gz" version="9.0.7"/>
          <condor_tarball arch="default" os="ubuntu20, linux-ubuntu20" tar_file="/tmp/condor-9.0.7-x86_64_Ubuntu20-stripped.tar.gz" version="9.0.7"/>
          <condor_tarball arch="default" os="default, rhel7, linux-rhel7" tar_file="/tmp/condor-9.0.16-x86_64_CentOS7-stripped.tar.gz" version="9.0.16,default"/>
          <condor_tarball arch="default" os="rhel8, linux-rhel8" tar_file="/tmp/condor-9.0.16-x86_64_CentOS8-stripped.tar.gz" version="9.0.16,default"/>
          <condor_tarball arch="default" os="ubuntu18, linux-ubuntu18" tar_file="/tmp/condor-9.0.16-x86_64_Ubuntu18-stripped.tar.gz" version="9.0.16,default"/>
          <condor_tarball arch="default" os="ubuntu20, linux-ubuntu20" tar_file="/tmp/condor-9.0.16-x86_64_Ubuntu20-stripped.tar.gz" version="9.0.16,default"/>
          <condor_tarball arch="default" os="default, rhel7, linux-rhel7" tar_file="/tmp/condor-9.0.18-x86_64_CentOS7-stripped.tar.gz" version="9.0.18,9.0.x"/>
          <condor_tarball arch="default" os="ubuntu18, linux-ubuntu18" tar_file="/tmp/condor-9.0.18-x86_64_Ubuntu18-stripped.tar.gz" version="9.0.18,9.0.x"/>
          <condor_tarball arch="default" os="ubuntu20, linux-ubuntu20" tar_file="/tmp/condor-9.0.18-x86_64_Ubuntu20-stripped.tar.gz" version="9.0.18,9.0.x"/>
          <condor_tarball arch="default" os="default, rhel7, linux-rhel7" tar_file="/tmp/condor-10.0.9-x86_64_CentOS7-stripped.tar.gz" version="10.0.9,10.0.x"/>
          <condor_tarball arch="default" os="rhel8, linux-rhel8" tar_file="/tmp/condor-10.0.9-x86_64_AlmaLinux8-stripped.tar.gz" version="10.0.9,10.0.x"/>
          <condor_tarball arch="default" os="ubuntu20, linux-ubuntu20" tar_file="/tmp/condor-10.0.9-x86_64_Ubuntu20-stripped.tar.gz" version="10.0.9,10.0.x"/>
          <condor_tarball arch="ppc64le" os="rhel8, linux-rhel8" tar_file="/tmp/condor-10.0.9-ppc64le_AlmaLinux8-stripped.tar.gz" version="10.0.9,10.0.x"/>
          <condor_tarball arch="aarch64" os="rhel8, linux-rhel8" tar_file="/tmp/condor-10.0.9-aarch64_AlmaLinux8-stripped.tar.gz" version="10.0.9,10.0.x"/>
          <condor_tarball arch="default" os="default, rhel7, linux-rhel7" tar_file="/tmp/condor-10.9.0-x86_64_CentOS7-stripped.tar.gz" version="10.9.0,10.x"/>
          <condor_tarball arch="default" os="rhel8, linux-rhel8" tar_file="/tmp/condor-10.9.0-x86_64_AlmaLinux8-stripped.tar.gz" version="10.9.0,10.x"/>
          <condor_tarball arch="default" os="ubuntu20, linux-ubuntu20" tar_file="/tmp/condor-10.9.0-x86_64_Ubuntu20-stripped.tar.gz" version="10.9.0,10.x"/>
          <condor_tarball arch="ppc64le" os="rhel8, linux-rhel8" tar_file="/tmp/condor-10.9.0-ppc64le_AlmaLinux8-stripped.tar.gz" version="10.9.0,10.x"/>
          <condor_tarball arch="aarch64" os="rhel8, linux-rhel8" tar_file="/tmp/condor-10.9.0-aarch64_AlmaLinux8-stripped.tar.gz" version="10.9.0,10.x"/>
          <condor_tarball arch="default" os="default, rhel7, linux-rhel7" tar_file="/tmp/condor-23.0.0-x86_64_CentOS7-stripped.tar.gz" version="23.0.0,23.0.x"/>
          <condor_tarball arch="default" os="rhel8, linux-rhel8" tar_file="/tmp/condor-23.0.0-x86_64_AlmaLinux8-stripped.tar.gz" version="23.0.0,23.0.x"/>
          <condor_tarball arch="default" os="ubuntu20, linux-ubuntu20" tar_file="/tmp/condor-23.0.0-x86_64_Ubuntu20-stripped.tar.gz" version="23.0.0,23.0.x"/>
          <condor_tarball arch="ppc64le" os="rhel8, linux-rhel8" tar_file="/tmp/condor-23.0.0-ppc64le_AlmaLinux8-stripped.tar.gz" version="23.0.0,23.0.x"/>
          <condor_tarball arch="aarch64" os="rhel8, linux-rhel8" tar_file="/tmp/condor-23.0.0-aarch64_AlmaLinux8-stripped.tar.gz" version="23.0.0,23.0.x"/>
       </condor_tarballs>
    </glidein>

Optionally, you can create a pre configuration hook to fully automate the tarballs download and setup.