Protocol API¶

CADC¶

Class to facilitate data transfer on CANFAR using the CADC tools.

CADCClient ¶

Class for CANFAR CADC API.

delete ¶

Python

delete(file, certfile=None, namespace='cadc:CHIMEFRB', verbose=0)

Delete a file from the CANFAR file server.

Parameters:

Name	Type	Description	Default
`file`	`List[str]`	List of source files to delete.	required
`certfile`	`Optional[str]`	Certificate. Defaults to None.	`None`
`namespace`	`str`	Minoc Namespace. Defaults to "cadc:CHIMEFRB".	`'cadc:CHIMEFRB'`
`verbose`	`int`	Verbosity level. Defaults to 0.	`0`

exist ¶

Python

exist(file, certfile=None, namespace='cadc:CHIMEFRB', verbose=0)

Check if a file exists.

Parameters:

Name	Type	Description	Default
`file`	`str`	File to check.	required
`certfile`	`Optional[str]`	Certificate. Defaults to None.	`None`
`namespace`	`str`	Minoc Namespace. Defaults to "cadc:CHIMEFRB".	`'cadc:CHIMEFRB'`
`verbose`	`int`	Verbosity. Defaults to 0.	`0`

Returns:

Name	Type	Description
`bool`		True if file exists.

Example:

Text Only

>>> exists("/data/chime/intensity/raw/2023/01/01/")

get ¶

Python

get(source, destination, certfile=None, namespace='cadc:CHIMEFRB', verbose=0)

Retrieve a file, stored on the CANFAR file server, and copy it locally.

Parameters:

Name	Type	Description	Default
`source`	`List[str]`	List of source files to retrieve.	required
`destination`	`List[str]`	List of destination files to copy to.	required
`certfile`	`Optional[str]`	Certificate. Defaults to None.	`None`
`namespace`	`str`	Minoc Namespace. Defaults to "cadc:CHIMEFRB".	`'cadc:CHIMEFRB'`
`verbose`	`int`	Verbosity level. Defaults to 0.	`0`

info ¶

Python

info(filename, certfile=None, namespace='cadc:CHIMEFRB', verbose=0)

Get the metadata for a list of files.

Parameters:

Name	Type	Description	Default
`filenames`	`List[str]`	List of filenames to get metadata for.	required
`certfile`	`Optional[str]`	Certificate. Defaults to None.	`None`
`namespace`	`_type_`	Minoc Namespace. Defaults to "cadc:CHIMEFRB".	`'cadc:CHIMEFRB'`
`aggregate`	`bool`	Aggregate the results. Defaults to False.	required
`verbose`	`int`	Verbosity level. Defaults to 0.	`0`

Returns:

Type	Description
`Tuple[Any, Any, Any, Any, Any, Any, Any]`	Tuple[Any]: Metadata for the files.

pdelete ¶

Python

pdelete(file, certfile=None, namespace='cadc:CHIMEFRB', processors=os.cpu_count() or 1, verbose=0)

Parallelly delete files from the CANFAR file server.

Parameters:

Name	Type	Description	Default
`file`	`List[str]`	List of source files to delete.	required
`certfile`	`Optional[str]`	Certificate. Defaults to None.	`None`
`namespace`	`_type_`	Minoc Namespace. Defaults to "cadc:CHIMEFRB".	`'cadc:CHIMEFRB'`
`processors`	`int`	Number of processes to use. Defaults to os.cpu_count() or 1.	`cpu_count() or 1`
`verbose`	`int`	Verbosity level. Defaults to 0.	`0`

pget ¶

Python

pget(source, destination, certfile=None, namespace='cadc:CHIMEFRB', processors=os.cpu_count() or 1, verbose=0)

Parallelly retrieve files, stored at CANFAR file server, and copy it locally.

Parameters:

Name	Type	Description	Default
`source`	`List[str]`	List of source files to retrieve.	required
`destination`	`List[str]`	List of destination files to copy to.	required
`certfile`	`Optional[str]`	Certificate. Defaults to None.	`None`
`namespace`	`_type_`	Minoc Namespace. Defaults to "cadc:CHIMEFRB".	`'cadc:CHIMEFRB'`
`processors`	`int`	Number of processes to use. Defaults to os.cpu_count() or 1.	`cpu_count() or 1`
`verbose`	`int`	Verbosity level. Defaults to 0.	`0`

pput ¶

Python

pput(source, destination, certfile=None, namespace='cadc:CHIMEFRB', overwrite=False, processors=os.cpu_count() or 1, verbose=0)

Parallelly transfer files, stored locally, to the CANFAR file server.

Parameters:

Name	Type	Description	Default
`source`	`List[str]`	List of source files to transfer.	required
`destination`	`List[str]`	List of destination files to transfer to.	required
`certfile`	`Optional[str]`	Certificate. Defaults to None.	`None`
`namespace`	`_type_`	Minoc Namespace. Defaults to "cadc:CHIMEFRB".	`'cadc:CHIMEFRB'`
`overwrite`	`bool`	Overwrite existing files.	`False`
`processors`	`int`	Number of processes to use. Defaults to os.cpu_count() or 1.	`cpu_count() or 1`
`verbose`	`int`	Verbosity level. Defaults to 0.	`0`

put ¶

Python

put(source, destination, certfile=None, namespace='cadc:CHIMEFRB', overwrite=False, verbose=0)

Transfer a file, stored locally, and copy it to the CANFAR file server.

Parameters:

Name	Type	Description	Default
`source`	`List[str]`	List of source files to transfer.	required
`destination`	`List[str]`	List of destination files to transfer to.	required
`certfile`	`Optional[str]`	Certificate. Defaults to None.	`None`
`namespace`	`str`	Minoc Namespace. Defaults to "cadc:CHIMEFRB".	`'cadc:CHIMEFRB'`
`overwrite`	`bool`	Overwrite existing files.	`False`
`verbose`	`int`	Verbosity level. Defaults to 0.	`0`

query ¶

Python

query(directory, certfile=None, namespace='cadc:CHIMEFRB', timeout=60, verbose=0)

Get list of files in a directory.

Parameters:

Name	Type	Description	Default
`directory`	`str`	Directory to get the size of.	required
`certfile`	`str`	Certificate file. Defaults to None.	`None`
`namespace`	`str`	Minoc Namespace. Defaults to "cadc:CHIMEFRB".	`'cadc:CHIMEFRB'`
`timeout`	`int`	Timeout. Defaults to 60.	`60`
`verbose`	`int`	Verbosity. Defaults to 0.	`0`

Returns:

Type	Description
`List[str]`	List[str]: List of files in the directory.

Example

size("/data/chime/intensity/raw/2023/01/01/")

size ¶

Python

size(directory, certfile=None, namespace='cadc:CHIMEFRB', timeout=60, verbose=0)

Get the size of a directory in GB.

Parameters:

Name	Type	Description	Default
`directory`	`str`	Directory to get the size of.	required
`certfile`	`Optional[str]`	Certificate. Defaults to None.	`None`
`namespace`	`_type_`	Minoc Namespace. Defaults to "cadc:CHIMEFRB".	`'cadc:CHIMEFRB'`
`timeout`	`int`	Timeout. Defaults to 60.	`60`
`verbose`	`int`	Verbosity level. Defaults to 0.	`0`

Returns:

Name	Type	Description
`float`	`float`	Size of the directory in GB.

Example

size("/data/chime/intensity/raw/2023/01/01/")

DillProcess ¶

Bases: Process

A Process class that uses dill to serialize the target function before execution.

Parameters:

Name	Type	Description	Default
`Process`	`object`	Python Process class.	required

init ¶

Python

__init__(*args, **kwargs)

Initialize the DillProcess class.

run ¶

Python

run()

Run the DillProcess.

split ¶

Python

split(data, count)

Split a list into batches.

Parameters:

Name	Type	Description	Default
`data`	`List[Any]`	List to split.	required
`count`	`int`	Number of batches to split into.	required

Returns:

Type	Description
`List[List[Any]]`	List[List[Any]]: List of batches.

POSIX¶

Implement POSIX protocols to do the following.

delete: delete a file on the storage element.
push: replicate a file to a remote destination.
pull: replicate a file from a remote location.
move: move a file from one local location to another.
copy: copy a file from one local location to another.

PosixApi ¶

Bases: Protocol

Class for POSIX API.

Attributes¶

Protocol : Protocol

Methods¶

delete(path: Union[str, Path]) Delete a file on the local storage element.

push(source: Union[str, Path], destination: Union[str, Path]) Push the file to the remote destination.

pull(source: Union[str, Path], destination: Union[str, Path]) Pull the file from the remote destination.

move(source, destination) Move a file from one local location to another.

copy(source, destination) Copy a file from one local location to another.

init ¶

Python

__init__(**kwargs)

Initialize the class.

copy ¶

Python

copy(source, destination)

Copy a file locally from one location to another.

delete ¶

Python

delete(path)

Delete a file on the storage element.

Parameters¶

path : Union[str, Path] Path for the file or directory to be deleted.

info ¶

Python

info(path)

Gather some info file about the file.

Parameters¶

path : Union[str, Path] Path to look for file at

mkdir ¶

Python

mkdir(path)

Make a directory on the local storage element.

Parameters¶

path : Union[str, Path] Path for the new directory.

move ¶

Python

move(source, destination)

Move a file locally from one location to another.

Parameters¶

source : Union[str, Path] Source path - local host. destination : Union[str, Path] Destination path - local host.

pull ¶

Python

pull(source, destination)

Pull the file from the remote destination.

Use rsync to pull the file from the remote destination.

Parameters¶

source : Union[str, Path] Source path - remote host. destination : Union[str, Path] Destination path - local host.

push ¶

Python

push(source, destination)

Push the file to the remote destination.

Use rsync to push the file to the remote destination.

Parameters¶

source : Union[str, Path] Source path - local host. destination : Union[str, Path] Destination path - remote host.

VOS¶

Implement VOSpace protocols to do the following.

delete: delete a file on the storage element.
push: replicate a file to a remote destination.
pull: replicate a file from a remote location.
move: move a file from one local location to another.
copy: copy a file from one local location to another.

VosApi ¶

Bases: Protocol

Class for VOSpace API.

init ¶

Python

__init__(**kwargs)

Initialize the VOSpace API.

chmod ¶

Python

chmod(node, mode)

Change global permissions of the file/directory.

connect ¶

Python

connect()

Connect to the VOSpace file server.

copy ¶

Python

copy(source, destination)

Copy file/directory within VOSpace.

delete ¶

Python

delete(node)

Delete the VOSpace node and everything in it.

exists ¶

Python

exists(node)

Check if the file/directory exists on VOSpace.

isdir ¶

Python

isdir(node)

Check if the node is a directory.

isfile ¶

Python

isfile(node)

Check if the node is a file.

make_readable_by_chime_frb_ro ¶

Python

make_readable_by_chime_frb_ro(node)

Make the node writable by all of chime frb group.

make_writable_by_chime_frb_admin ¶

Python

make_writable_by_chime_frb_admin(node)

Make the node writable by chime frb admin group.

make_writable_by_chime_frb_rw ¶

Python

make_writable_by_chime_frb_rw(node)

Make the node writable by all of chime frb group.

mkdir ¶

Python

mkdir(node)

Create a directory on VOSpace.

move ¶

Python

move(source, destination)

Move file/directory within VOSpace.

pull ¶

Python

pull(source, destination)

Pull the file/directory from VOSpace to local space.

push ¶

Python

push(source, destination, n_streams=5, cache_path=None, verbose=False)

Push the file/directory to VOSpace.

sync ¶

Python

sync(source, destination)

Sync the directory structure within VOSpace.

Protocol API¶

CADC¶

CADCClient ¶

delete ¶

exist ¶

get ¶

info ¶

pdelete ¶

pget ¶

pput ¶

put ¶

query ¶

size ¶

DillProcess ¶

__init__ ¶

run ¶

split ¶

POSIX¶

PosixApi ¶

Attributes¶

Methods¶

__init__ ¶

copy ¶

delete ¶

Parameters¶

info ¶

Parameters¶

mkdir ¶

Parameters¶

move ¶

Parameters¶

pull ¶

Parameters¶

push ¶

Parameters¶

VOS¶

VosApi ¶

__init__ ¶

chmod ¶

connect ¶

copy ¶

delete ¶

exists ¶

isdir ¶

isfile ¶

make_readable_by_chime_frb_ro ¶

make_writable_by_chime_frb_admin ¶

make_writable_by_chime_frb_rw ¶

mkdir ¶

move ¶

pull ¶

push ¶

sync ¶

init ¶

init ¶

init ¶