Protocol API¶
CADC¶
Class to facilitate data transfer on CANFAR using the CADC tools.
CADCClient
¶
Class for CANFAR CADC API.
delete
¶
Delete a file from the CANFAR file server.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
file
|
List[str]
|
List of source files to delete. |
required |
certfile
|
Optional[str]
|
Certificate. Defaults to None. |
None
|
namespace
|
str
|
Minoc Namespace. Defaults to "cadc:CHIMEFRB". |
'cadc:CHIMEFRB'
|
verbose
|
int
|
Verbosity level. Defaults to 0. |
0
|
exist
¶
Check if a file exists.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
file
|
str
|
File to check. |
required |
certfile
|
Optional[str]
|
Certificate. Defaults to None. |
None
|
namespace
|
str
|
Minoc Namespace. Defaults to "cadc:CHIMEFRB". |
'cadc:CHIMEFRB'
|
verbose
|
int
|
Verbosity. Defaults to 0. |
0
|
Returns:
Name | Type | Description |
---|---|---|
bool |
True if file exists. |
Example:
>>> exists("/data/chime/intensity/raw/2023/01/01/")
get
¶
Retrieve a file, stored on the CANFAR file server, and copy it locally.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
source
|
List[str]
|
List of source files to retrieve. |
required |
destination
|
List[str]
|
List of destination files to copy to. |
required |
certfile
|
Optional[str]
|
Certificate. Defaults to None. |
None
|
namespace
|
str
|
Minoc Namespace. Defaults to "cadc:CHIMEFRB". |
'cadc:CHIMEFRB'
|
verbose
|
int
|
Verbosity level. Defaults to 0. |
0
|
info
¶
Get the metadata for a list of files.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
filenames
|
List[str]
|
List of filenames to get metadata for. |
required |
certfile
|
Optional[str]
|
Certificate. Defaults to None. |
None
|
namespace
|
_type_
|
Minoc Namespace. Defaults to "cadc:CHIMEFRB". |
'cadc:CHIMEFRB'
|
aggregate
|
bool
|
Aggregate the results. Defaults to False. |
required |
verbose
|
int
|
Verbosity level. Defaults to 0. |
0
|
Returns:
Type | Description |
---|---|
Tuple[Any, Any, Any, Any, Any, Any, Any]
|
Tuple[Any]: Metadata for the files. |
pdelete
¶
pdelete(file, certfile=None, namespace='cadc:CHIMEFRB', processors=os.cpu_count() or 1, verbose=0)
Parallelly delete files from the CANFAR file server.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
file
|
List[str]
|
List of source files to delete. |
required |
certfile
|
Optional[str]
|
Certificate. Defaults to None. |
None
|
namespace
|
_type_
|
Minoc Namespace. Defaults to "cadc:CHIMEFRB". |
'cadc:CHIMEFRB'
|
processors
|
int
|
Number of processes to use. Defaults to os.cpu_count() or 1. |
cpu_count() or 1
|
verbose
|
int
|
Verbosity level. Defaults to 0. |
0
|
pget
¶
pget(source, destination, certfile=None, namespace='cadc:CHIMEFRB', processors=os.cpu_count() or 1, verbose=0)
Parallelly retrieve files, stored at CANFAR file server, and copy it locally.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
source
|
List[str]
|
List of source files to retrieve. |
required |
destination
|
List[str]
|
List of destination files to copy to. |
required |
certfile
|
Optional[str]
|
Certificate. Defaults to None. |
None
|
namespace
|
_type_
|
Minoc Namespace. Defaults to "cadc:CHIMEFRB". |
'cadc:CHIMEFRB'
|
processors
|
int
|
Number of processes to use. Defaults to os.cpu_count() or 1. |
cpu_count() or 1
|
verbose
|
int
|
Verbosity level. Defaults to 0. |
0
|
pput
¶
pput(source, destination, certfile=None, namespace='cadc:CHIMEFRB', overwrite=False, processors=os.cpu_count() or 1, verbose=0)
Parallelly transfer files, stored locally, to the CANFAR file server.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
source
|
List[str]
|
List of source files to transfer. |
required |
destination
|
List[str]
|
List of destination files to transfer to. |
required |
certfile
|
Optional[str]
|
Certificate. Defaults to None. |
None
|
namespace
|
_type_
|
Minoc Namespace. Defaults to "cadc:CHIMEFRB". |
'cadc:CHIMEFRB'
|
overwrite
|
bool
|
Overwrite existing files. |
False
|
processors
|
int
|
Number of processes to use. Defaults to os.cpu_count() or 1. |
cpu_count() or 1
|
verbose
|
int
|
Verbosity level. Defaults to 0. |
0
|
put
¶
put(source, destination, certfile=None, namespace='cadc:CHIMEFRB', overwrite=False, verbose=0)
Transfer a file, stored locally, and copy it to the CANFAR file server.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
source
|
List[str]
|
List of source files to transfer. |
required |
destination
|
List[str]
|
List of destination files to transfer to. |
required |
certfile
|
Optional[str]
|
Certificate. Defaults to None. |
None
|
namespace
|
str
|
Minoc Namespace. Defaults to "cadc:CHIMEFRB". |
'cadc:CHIMEFRB'
|
overwrite
|
bool
|
Overwrite existing files. |
False
|
verbose
|
int
|
Verbosity level. Defaults to 0. |
0
|
query
¶
Get list of files in a directory.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
directory
|
str
|
Directory to get the size of. |
required |
certfile
|
str
|
Certificate file. Defaults to None. |
None
|
namespace
|
str
|
Minoc Namespace. Defaults to "cadc:CHIMEFRB". |
'cadc:CHIMEFRB'
|
timeout
|
int
|
Timeout. Defaults to 60. |
60
|
verbose
|
int
|
Verbosity. Defaults to 0. |
0
|
Returns:
Type | Description |
---|---|
List[str]
|
List[str]: List of files in the directory. |
Example
size("/data/chime/intensity/raw/2023/01/01/")
size
¶
Get the size of a directory in GB.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
directory
|
str
|
Directory to get the size of. |
required |
certfile
|
Optional[str]
|
Certificate. Defaults to None. |
None
|
namespace
|
_type_
|
Minoc Namespace. Defaults to "cadc:CHIMEFRB". |
'cadc:CHIMEFRB'
|
timeout
|
int
|
Timeout. Defaults to 60. |
60
|
verbose
|
int
|
Verbosity level. Defaults to 0. |
0
|
Returns:
Name | Type | Description |
---|---|---|
float |
float
|
Size of the directory in GB. |
Example
size("/data/chime/intensity/raw/2023/01/01/")
DillProcess
¶
Bases: Process
A Process class that uses dill to serialize the target function before execution.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
Process
|
object
|
Python Process class. |
required |
split
¶
Split a list into batches.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
data
|
List[Any]
|
List to split. |
required |
count
|
int
|
Number of batches to split into. |
required |
Returns:
Type | Description |
---|---|
List[List[Any]]
|
List[List[Any]]: List of batches. |
POSIX¶
Implement POSIX protocols to do the following.
- delete: delete a file on the storage element.
- push: replicate a file to a remote destination.
- pull: replicate a file from a remote location.
- move: move a file from one local location to another.
- copy: copy a file from one local location to another.
PosixApi
¶
Bases: Protocol
Class for POSIX API.
Attributes¶
Protocol : Protocol
Methods¶
delete(path: Union[str, Path]) Delete a file on the local storage element.
push(source: Union[str, Path], destination: Union[str, Path]) Push the file to the remote destination.
pull(source: Union[str, Path], destination: Union[str, Path]) Pull the file from the remote destination.
move(source, destination) Move a file from one local location to another.
copy(source, destination) Copy a file from one local location to another.
delete
¶
Delete a file on the storage element.
Parameters¶
path : Union[str, Path] Path for the file or directory to be deleted.
info
¶
mkdir
¶
Make a directory on the local storage element.
Parameters¶
path : Union[str, Path] Path for the new directory.
move
¶
Move a file locally from one location to another.
Parameters¶
source : Union[str, Path] Source path - local host. destination : Union[str, Path] Destination path - local host.
VOS¶
Implement VOSpace protocols to do the following.
- delete: delete a file on the storage element.
- push: replicate a file to a remote destination.
- pull: replicate a file from a remote location.
- move: move a file from one local location to another.
- copy: copy a file from one local location to another.
VosApi
¶
Bases: Protocol
Class for VOSpace API.