Skip to content

Protocol API

CADC

Class to facilitate data transfer on CANFAR using the CADC tools.

CADCClient

Class for CANFAR CADC API.

delete

Python
delete(file, certfile=None, namespace='cadc:CHIMEFRB', verbose=0)

Delete a file from the CANFAR file server.

Parameters:

Name Type Description Default
file List[str]

List of source files to delete.

required
certfile Optional[str]

Certificate. Defaults to None.

None
namespace str

Minoc Namespace. Defaults to "cadc:CHIMEFRB".

'cadc:CHIMEFRB'
verbose int

Verbosity level. Defaults to 0.

0

exist

Python
exist(file, certfile=None, namespace='cadc:CHIMEFRB', verbose=0)

Check if a file exists.

Parameters:

Name Type Description Default
file str

File to check.

required
certfile Optional[str]

Certificate. Defaults to None.

None
namespace str

Minoc Namespace. Defaults to "cadc:CHIMEFRB".

'cadc:CHIMEFRB'
verbose int

Verbosity. Defaults to 0.

0

Returns:

Name Type Description
bool

True if file exists.

Example:

Text Only
>>> exists("/data/chime/intensity/raw/2023/01/01/")

get

Python
get(source, destination, certfile=None, namespace='cadc:CHIMEFRB', verbose=0)

Retrieve a file, stored on the CANFAR file server, and copy it locally.

Parameters:

Name Type Description Default
source List[str]

List of source files to retrieve.

required
destination List[str]

List of destination files to copy to.

required
certfile Optional[str]

Certificate. Defaults to None.

None
namespace str

Minoc Namespace. Defaults to "cadc:CHIMEFRB".

'cadc:CHIMEFRB'
verbose int

Verbosity level. Defaults to 0.

0

info

Python
info(filename, certfile=None, namespace='cadc:CHIMEFRB', verbose=0)

Get the metadata for a list of files.

Parameters:

Name Type Description Default
filenames List[str]

List of filenames to get metadata for.

required
certfile Optional[str]

Certificate. Defaults to None.

None
namespace _type_

Minoc Namespace. Defaults to "cadc:CHIMEFRB".

'cadc:CHIMEFRB'
aggregate bool

Aggregate the results. Defaults to False.

required
verbose int

Verbosity level. Defaults to 0.

0

Returns:

Type Description
Tuple[Any, Any, Any, Any, Any, Any, Any]

Tuple[Any]: Metadata for the files.

pdelete

Python
pdelete(file, certfile=None, namespace='cadc:CHIMEFRB', processors=os.cpu_count() or 1, verbose=0)

Parallelly delete files from the CANFAR file server.

Parameters:

Name Type Description Default
file List[str]

List of source files to delete.

required
certfile Optional[str]

Certificate. Defaults to None.

None
namespace _type_

Minoc Namespace. Defaults to "cadc:CHIMEFRB".

'cadc:CHIMEFRB'
processors int

Number of processes to use. Defaults to os.cpu_count() or 1.

cpu_count() or 1
verbose int

Verbosity level. Defaults to 0.

0

pget

Python
pget(source, destination, certfile=None, namespace='cadc:CHIMEFRB', processors=os.cpu_count() or 1, verbose=0)

Parallelly retrieve files, stored at CANFAR file server, and copy it locally.

Parameters:

Name Type Description Default
source List[str]

List of source files to retrieve.

required
destination List[str]

List of destination files to copy to.

required
certfile Optional[str]

Certificate. Defaults to None.

None
namespace _type_

Minoc Namespace. Defaults to "cadc:CHIMEFRB".

'cadc:CHIMEFRB'
processors int

Number of processes to use. Defaults to os.cpu_count() or 1.

cpu_count() or 1
verbose int

Verbosity level. Defaults to 0.

0

pput

Python
pput(source, destination, certfile=None, namespace='cadc:CHIMEFRB', overwrite=False, processors=os.cpu_count() or 1, verbose=0)

Parallelly transfer files, stored locally, to the CANFAR file server.

Parameters:

Name Type Description Default
source List[str]

List of source files to transfer.

required
destination List[str]

List of destination files to transfer to.

required
certfile Optional[str]

Certificate. Defaults to None.

None
namespace _type_

Minoc Namespace. Defaults to "cadc:CHIMEFRB".

'cadc:CHIMEFRB'
overwrite bool

Overwrite existing files.

False
processors int

Number of processes to use. Defaults to os.cpu_count() or 1.

cpu_count() or 1
verbose int

Verbosity level. Defaults to 0.

0

put

Python
put(source, destination, certfile=None, namespace='cadc:CHIMEFRB', overwrite=False, verbose=0)

Transfer a file, stored locally, and copy it to the CANFAR file server.

Parameters:

Name Type Description Default
source List[str]

List of source files to transfer.

required
destination List[str]

List of destination files to transfer to.

required
certfile Optional[str]

Certificate. Defaults to None.

None
namespace str

Minoc Namespace. Defaults to "cadc:CHIMEFRB".

'cadc:CHIMEFRB'
overwrite bool

Overwrite existing files.

False
verbose int

Verbosity level. Defaults to 0.

0

query

Python
query(directory, certfile=None, namespace='cadc:CHIMEFRB', timeout=60, verbose=0)

Get list of files in a directory.

Parameters:

Name Type Description Default
directory str

Directory to get the size of.

required
certfile str

Certificate file. Defaults to None.

None
namespace str

Minoc Namespace. Defaults to "cadc:CHIMEFRB".

'cadc:CHIMEFRB'
timeout int

Timeout. Defaults to 60.

60
verbose int

Verbosity. Defaults to 0.

0

Returns:

Type Description
List[str]

List[str]: List of files in the directory.

Example

size("/data/chime/intensity/raw/2023/01/01/")

size

Python
size(directory, certfile=None, namespace='cadc:CHIMEFRB', timeout=60, verbose=0)

Get the size of a directory in GB.

Parameters:

Name Type Description Default
directory str

Directory to get the size of.

required
certfile Optional[str]

Certificate. Defaults to None.

None
namespace _type_

Minoc Namespace. Defaults to "cadc:CHIMEFRB".

'cadc:CHIMEFRB'
timeout int

Timeout. Defaults to 60.

60
verbose int

Verbosity level. Defaults to 0.

0

Returns:

Name Type Description
float float

Size of the directory in GB.

Example

size("/data/chime/intensity/raw/2023/01/01/")

DillProcess

Bases: Process

A Process class that uses dill to serialize the target function before execution.

Parameters:

Name Type Description Default
Process object

Python Process class.

required

__init__

Python
__init__(*args, **kwargs)

Initialize the DillProcess class.

run

Python
run()

Run the DillProcess.

split

Python
split(data, count)

Split a list into batches.

Parameters:

Name Type Description Default
data List[Any]

List to split.

required
count int

Number of batches to split into.

required

Returns:

Type Description
List[List[Any]]

List[List[Any]]: List of batches.

POSIX

Implement POSIX protocols to do the following.

  1. delete: delete a file on the storage element.
  2. push: replicate a file to a remote destination.
  3. pull: replicate a file from a remote location.
  4. move: move a file from one local location to another.
  5. copy: copy a file from one local location to another.

PosixApi

Bases: Protocol

Class for POSIX API.

Attributes

Protocol : Protocol

Methods

delete(path: Union[str, Path]) Delete a file on the local storage element.

push(source: Union[str, Path], destination: Union[str, Path]) Push the file to the remote destination.

pull(source: Union[str, Path], destination: Union[str, Path]) Pull the file from the remote destination.

move(source, destination) Move a file from one local location to another.

copy(source, destination) Copy a file from one local location to another.

__init__

Python
__init__(**kwargs)

Initialize the class.

copy

Python
copy(source, destination)

Copy a file locally from one location to another.

delete

Python
delete(path)

Delete a file on the storage element.

Parameters

path : Union[str, Path] Path for the file or directory to be deleted.

info

Python
info(path)

Gather some info file about the file.

Parameters

path : Union[str, Path] Path to look for file at

mkdir

Python
mkdir(path)

Make a directory on the local storage element.

Parameters

path : Union[str, Path] Path for the new directory.

move

Python
move(source, destination)

Move a file locally from one location to another.

Parameters

source : Union[str, Path] Source path - local host. destination : Union[str, Path] Destination path - local host.

pull

Python
pull(source, destination)

Pull the file from the remote destination.

Use rsync to pull the file from the remote destination.

Parameters

source : Union[str, Path] Source path - remote host. destination : Union[str, Path] Destination path - local host.

push

Python
push(source, destination)

Push the file to the remote destination.

Use rsync to push the file to the remote destination.

Parameters

source : Union[str, Path] Source path - local host. destination : Union[str, Path] Destination path - remote host.

VOS

Implement VOSpace protocols to do the following.

  1. delete: delete a file on the storage element.
  2. push: replicate a file to a remote destination.
  3. pull: replicate a file from a remote location.
  4. move: move a file from one local location to another.
  5. copy: copy a file from one local location to another.

VosApi

Bases: Protocol

Class for VOSpace API.

__init__

Python
__init__(**kwargs)

Initialize the VOSpace API.

chmod

Python
chmod(node, mode)

Change global permissions of the file/directory.

connect

Python
connect()

Connect to the VOSpace file server.

copy

Python
copy(source, destination)

Copy file/directory within VOSpace.

delete

Python
delete(node)

Delete the VOSpace node and everything in it.

exists

Python
exists(node)

Check if the file/directory exists on VOSpace.

isdir

Python
isdir(node)

Check if the node is a directory.

isfile

Python
isfile(node)

Check if the node is a file.

make_readable_by_chime_frb_ro

Python
make_readable_by_chime_frb_ro(node)

Make the node writable by all of chime frb group.

make_writable_by_chime_frb_admin

Python
make_writable_by_chime_frb_admin(node)

Make the node writable by chime frb admin group.

make_writable_by_chime_frb_rw

Python
make_writable_by_chime_frb_rw(node)

Make the node writable by all of chime frb group.

mkdir

Python
mkdir(node)

Create a directory on VOSpace.

move

Python
move(source, destination)

Move file/directory within VOSpace.

pull

Python
pull(source, destination)

Pull the file/directory from VOSpace to local space.

push

Python
push(source, destination, n_streams=5, cache_path=None, verbose=False)

Push the file/directory to VOSpace.

sync

Python
sync(source, destination)

Sync the directory structure within VOSpace.