Skip to main content

Access Cloud Object Storage (COS)

Except for using Hyper File System (HFS) as the compute storage, TWSC TWNIA2 (HPC CLI) can also use TWSC Cloud Object Storage (COS) as a storage option for data accessed less frequently.

This article will demonstrate two method to access files in TWSC Cloud Object Storage (COS) from TWNIA2 (HPC CLI):

  1. S3 client tools
    With the S3 client tools, you can upload and download files via HTTPS (HyperText Transfer Protocol Secure) on the login node and when you submit jobs to the compute nodes.

  2. Mount COS as a network hard drive
    Mount COS as the storage space using FUSE (Filesystem in Userspace). Only the login node (ln01.twcc.ai) is available.

info

It is recommended to use S3 client tools:

  • The COS mounted via FUSE is not a local drive. To prevent data loss, you can only upload one file at a time (read and download multiple files is available).
  • There are also many other limitations, so mounting COS as a network hard drive is only available on the login node.

Below we demonstrate accessing COS with the two methods. You can choose either of them according to your needs.


1. S3 client tools​

There are many types of S3 client tools. The tools introduced below are mainly CLI tools. Choose a tool you preferred. In the following examples we use TWCC CLI and S3cmd to access COS data:

  • S3cmd
    Written in Python, S3md is one of the most well-known S3 CLI tools, with early development as well as complete and stable functionality.
  • TWCC CLI
    Consider using TWCC CLI if you are already familiar with it. You can use it to operate various TWSC services.
  • Other tools
    • Rclone
      Rclone is able to access to various cloud storage. It is well-suited for transferring small files between TWSC TWNIA2 (HPC CLI) and COS.
    • S5cmd
      S5cmd is written in Go language, and its transmission speed is faster than the tool written in Python but with fewer functions.

Installtion and Settings​

  • Accessing TWSC COS required the Access Key and the Secret Key. Please login to TWSC portal and go to the COS page to get the keys.
  • Go to the Third-party Software page of Cloud Object Storage (Shared storage space between project members) or Private COS (private storage space) to get the keys.


S3cmd​

  • Install
# Using pip to install
pip install s3cmd --user
  • Enter key
# Here we use vim as the editing tool, you may use your familiar editor to edit ~/.s3cfg
vim ~/.s3cfg
# Edit content
-----------------------
# Setup endpoint
host_base = cos.twcc.ai
host_bucket = cos.twcc.ai
use_https = True

# Setup access keys
access_key = <COS Access Key>
secret_key = <COS Secret Key>
-----------------------

TWCC CLI​

  • Install
# Using pip3 to install
pip3 install TWCC-CLI --user
  • For how to enter key, select projects or enter TWCC CLI environment, please refer to TWCC CLI document for more information.

Examples​

We demonstrate each example using S3cmd and TWCC CLI in order.

  • Create a bucket named mytwccbucket
# for S3cmd
s3cmd mb s3://mytwccbucket

# for TWCC CLI
twccli mk cos -bkt mytwccbucket
  • Upload files

Create an empty folder in /home directory named myfile.

touch ~/myfile

Copy the file to mytwccbucket bucket.

# for S3cmd
s3cmd put ~/myfile s3://mytwccbucket/

# for TWCC CLI
twccli cp cos -bkt mytwccbucket -fn myfile -sync to-cos
  • View files

Check if the upload is successful.

# for S3cmd
s3cmd ls s3://mytwccbucket/myfile

# for TWCC CLI
twccli ls cos -bkt mytwccbucket
  • Delete files

Delete myfile just uploaded.

# for S3cmd
s3cmd rm s3://mytwccbucket/myfile

# for TWCC CLI
twccli rm cos -bkt mytwccbucket -okey myfile
  • Delete buckets

Please make sure the bucket is empty before deleting it.

# for S3cmd
s3cmd rb s3://mytwccbucket

# for TWCC CLI
twccli rm cos -bkt mytwccbucket
info

Other parameters and description:

  • S3cmd Check by entering the s3cmd --help command or learn more on s3cmd GitHub.
  • TWCC CLI Check by entering the twccli --help command or learn more on TWCC CLI document.

2. Mount​

The following are two mounting tools: Goofys and S3fs, you may choose according to your needs.

info

The mounting methods are only able to used on ln01.twcc.ai login node.


Create a COS bucket​

  • Please refer to TWSC Cloud Object Storage manual and create a COS bucket on the TWSC portal (The management of buckets cannot be operated using TWNIA2).
  • After logging in to ln01.twcc.ai, create a ~/mount_cos folder under your account:
mkdir -p ~/mount_cos

Enter keys​

Create a folder under the ~/.aws directory, and copy TWSC COS keys to ~/.aws/credentials.

Here we use vim as the editing tool, you may use your familiar editor.

mkdir -p ~/.aws
vim ~/.aws/credentials

Enter the content below:

[default]
aws_access_key_id = COS Access Key
aws_secret_access_key = COS Secret Key

Goofys​

  • Written in Go language, the performance is good, so you can give priority to using it.

Mount command:

goofys --endpoint https://cos.twcc.ai mytwccbucket ~/mount_cos

S3fs​

  • A well-known s3 mounting tool.

Mount command:

s3fs mytwccbucket ~/mount_s3 -o url=https://cos.twcc.ai/ -o use_path_request_style

Access files​

After you complete mounting COS, you can access files in the newly created bucket mytwccbucket under the ~/mount_cos directory.

Example:

# Create a new file named myfile
touch ~/mount_cos/myfile

Now you can click on mytwccbucket on the Cloud Object Storage Management page on the TWCC portal. You can see that myfile has been added to the content.


Unmount​

Please exit the mounted folder and back to the /home directory.

cd ~/

Enter the unmount command.

fusermount -u ~/mount_cos

Enter the command below. The unmount is completed if there are no files listed. 

ls ~/mount_cos