sdss-access Reference¶

Path¶

exception sdss_access.path.path.AccessError[source]¶: Bases: Exception

class sdss_access.path.path.BasePath(pathfile, mirror=False, public=False, verbose=False)[source]¶

Bases: object

Class for construction of paths in general.

Variables:	templates (dict) – The set of templates read from the configuration file.

any(filetype, **kwargs)[source]¶

Checks if the local directory contains any of the type of file

Parameters:	filetype (str) – File type parameter.
Returns:	any (bool) – Boolean indicating if the any files exist in the expanded path on disk.

dir(filetype, **kwargs)[source]¶

Return the directory containing a file of a given type.

Parameters:	filetype (str) – File type parameter.
Returns:	dir (str) – Directory containing the file.

exists(filetype, **kwargs)[source]¶

Checks if the given type of file exists

Parameters:	filetype (str) – File type parameter.
Returns:	exists (bool) – Boolean indicating if the file exists on disk.

expand(filetype, **kwargs)[source]¶

Expand a wildcard path locally

Parameters:	filetype (str) – File type parameter. as_url (bool) – Boolean to return SAS urls refine (str) – Regular expression string to filter the list of files by before random selection
Returns:	expand (list) – List of expanded full paths of the given type.

full(filetype, **kwargs)[source]¶

Return the full name of a given type of file.

Parameters:	filetype (str) – File type parameter.
Returns:	full (str) – The full path to the file.

get_remote_base(scheme='https')[source]¶

location(filetype, base_dir=None, **kwargs)[source]¶

Return the location of the relative sas path of a given type of file.

Parameters:	filetype (str) – File type parameter.
Returns:	full (str) – The relative sas path to the file.

lookup_keys(name)[source]¶

Lookup the keyword arguments needed for a given path name

Parameters:	name (str) – The name of the path
Returns:	A list of keywords needed for filepath generation

name(filetype, **kwargs)[source]¶

Return the directory containing a file of a given type.

Parameters:	filetype (str) – File type parameter.
Returns:	name (str) – Name of a file with no directory information.

one(filetype, **kwargs)[source]¶

Returns random one of the given type of file

Parameters:	filetype (str) – File type parameter. as_url (bool) – Boolean to return SAS urls refine (str) – Regular expression string to filter the list of files by before random selection
Returns:	one (str) – Random file selected from the expanded list of full paths on disk.

random(filetype, **kwargs)[source]¶

Returns random number of the given type of file

Parameters:	filetype (str) – File type parameter. num (int) – The number of files to return as_url (bool) – Boolean to return SAS urls refine (str) – Regular expression string to filter the list of files by before random selection
Returns:	random (list) – Random file selected from the expanded list of full paths on disk.

refine(filelist, regex, filterdir='out', **kwargs)[source]¶

Returns a list of files filterd by a regular expression

Parameters:	filelist (list) – A list of files to filter on. regex (str) – The regular expression string to filter your list filterdir ({'in', 'out'}) – Indicates the filter to be inclusive or exclusive ‘out’ removes the items satisfying the regular expression ‘in’ keeps the items satisfying the regular expression
Returns:	refine (list) – A file list refined by an input regular expression.

set_base_dir(base_dir=None)[source]¶

set_netloc(netloc=None, sdss=None, dtn=None)[source]¶

set_remote_base(scheme=None)[source]¶

url(filetype, base_dir=None, sasdir='sas', **kwargs)[source]¶

Return the url of a given type of file.

Parameters:	filetype (str) – File type parameter.
Returns:	full (str) – The sas url to the file.

class sdss_access.path.path.Path(mirror=False, public=False, verbose=False)[source]¶

Bases: sdss_access.path.path.BasePath

Class for construction of paths in general. Sets a particular template file.

definitiondir(filetype, **kwargs)[source]¶

Returns definition subdirectory in PLATELIST_DIR of the form: NNNNXX.

Parameters:	filetype (str) – File type parameter. designid (int or str) – Design ID number. Will be converted to int internally.
Returns:	definitiondir (str) – Definition directory in the format `NNNNXX`.

platedir(filetype, **kwargs)[source]¶

Returns plate subdirectory in PLATELIST_DIR of the form: NNNNXX/NNNNNN.

Parameters:	filetype (str) – File type parameter. plateid (int or str) – Plate ID number. Will be converted to int internally.
Returns:	platedir (str) – Plate directory in the format `NNNNXX/NNNNNN`.

plateid6(filetype, **kwargs)[source]¶

Print plate ID, accounting for 5-6 digit plate IDs.

Parameters:	filetype (str) – File type parameter. plateid (int or str) – Plate ID number. Will be converted to int internally.
Returns:	plateid6 (str) – Plate ID formatted to a string of 6 characters.

spectrodir(filetype, **kwargs)[source]¶

Returns SPECTRO_REDUX or BOSS_SPECTRO_REDUX depending on the value of run2d.

Parameters:	filetype (str) – File type parameter. run2d (int or str) – 2D Reduction ID.
Returns:	spectrodir (str) – Value of the appropriate environment variable.

Sync¶

Auth¶

class sdss_access.sync.auth.Auth(netloc=None, public=False, verbose=False)[source]¶

Bases: object

load()[source]¶

ready()[source]¶

set_netloc(netloc=None)[source]¶

set_netrc()[source]¶

set_password(password=None, inquire=False)[source]¶

set_username(username=None, inquire=False)[source]¶

Client¶

exception sdss_access.sync.cli.CliError[source]¶: Bases: Exception

class sdss_access.sync.cli.Cli(label=None, data_dir=None, verbose=False)[source]¶

Bases: object

Class for providing command line interface (cli) sync scripts, and logs to local disk

foreground_run(command, test=False, logger=None, logall=False, message=None, outname=None, errname=None)[source]¶

A convenient wrapper to log and perform system calls.

Parameters:

Parameters:	command (str) – The system command to run. It will be split internally by shlex.split(). test (bool, optional) – Set this to true to not actually run the commands. logger (logging.logging, optional) – If passed a logging object, diagnostic output will be set to this object. logall (bool, optional) – Set this to true to always log stdout and stderr (at level DEBUG). Otherwise stdout and stderr will only be logged for nonzero status message (str, optional) – Call logger.critical(message) with this message and then call sys.exit(status). outname (str, optional) – If set, use `outname` as the name of a file to contain stdout. It is the responsibility of the caller of this function to clean up the resulting file. Otherwise a temporary file will be used. errname (str, optional) – Same as `outname`, but will contain stderr.
Returns:	(status,out,err) (tuple) – The exit status, stdout and stderr of the process

command (str) – The system command to run. It will be split internally by shlex.split().
test (bool, optional) – Set this to true to not actually run the commands.
logger (logging.logging, optional) – If passed a logging object, diagnostic output will be set to this object.
logall (bool, optional) – Set this to true to always log stdout and stderr (at level DEBUG). Otherwise stdout and stderr will only be logged for nonzero status
message (str, optional) – Call logger.critical(message) with this message and then call sys.exit(status).
outname (str, optional) – If set, use outname as the name of a file to contain stdout. It is the responsibility of the caller of this function to clean up the resulting file. Otherwise a temporary file will be used.
errname (str, optional) – Same as outname, but will contain stderr.

Returns:

(status,out,err) (tuple) – The exit status, stdout and stderr of the process

Examples

>>> status,out,err = transfer.common.system_call('date')

get_background_process(command=None, logfile=None, errfile=None, pause=1)[source]¶

get_path(index=None)[source]¶

set_dir()[source]¶

wait_for_processes(processes, pause=60)[source]¶

write_lines(path=None, lines=None)[source]¶

tmp_dir = '/tmp'¶

Http¶

class sdss_access.sync.http.HttpAccess(verbose=False)[source]¶

Bases: sdss_access.path.path.Path

Class for providing HTTP access via urllib.request (python3) or urllib2 (python2) to SDSS SAS Paths

download_url_to_path(url, path, force=False)[source]¶

Download a file from url via http, and put it at path

Parameters:	url (str) – URL of file to download path (str) – local path to put file in

get(filetype, **kwargs)[source]¶

Returns file name, downloading if remote access configured.

Parameters:	filetype (str) – type of file arguments (keyword) – keywords to fully specify path

Notes

Path templates are defined in $DIMAGE_DIR/data/dimage_paths.ini

local()[source]¶: Configures back to local access

remote(remote_base=None, username=None, password=None)[source]¶

Configures remote access

Parameters:	remote_base (str) – base URL path for remote repository username (str) – user name for remote repository password (str) – password for local repository

set_auth(username=None, password=None)[source]¶

Rsync¶

class sdss_access.sync.rsync.RsyncAccess(label='sdss_rsync', stream_count=5, mirror=False, public=False, verbose=False)[source]¶

Bases: sdss_access.path.path.Path

Class for providing Rsync access to SDSS SAS Paths

add(filetype, **kwargs)[source]¶: Adds a filepath into the list of tasks to download

commit(offset=None, limit=None, dryrun=False)[source]¶: Start the rsync download

generate_stream_task(task=None, out=None)[source]¶

get_locations(offset=None, limit=None)[source]¶

get_paths(offset=None, limit=None)[source]¶

get_stream()[source]¶

get_task_out(task=None)[source]¶

get_urls(offset=None, limit=None)[source]¶

refine_task(regex=None)[source]¶

remote(username=None, password=None, inquire=None)[source]¶: Configures remote access

reset()[source]¶

set_auth(username=None, password=None, inquire=True)[source]¶

set_stream()[source]¶: Sets the download streams

set_stream_task(task=None)[source]¶

shuffle()[source]¶

Stream¶

class sdss_access.sync.stream.Stream(stream_count=None, verbose=False)[source]¶

Bases: object

append_streamlet(index=None, task=None)[source]¶

append_task(location=None, source=None, destination=None)[source]¶

append_tasks_to_streamlets(offset=None, limit=None)[source]¶

commit_streamlet(streamlet=None)[source]¶

commit_streamlets()[source]¶

get_locations(offset=None, limit=None)[source]¶

get_streamlet(index=None, increment=1)[source]¶

refine_task(regex=None)[source]¶

reset()[source]¶

reset_streamlet()[source]¶

reset_task()[source]¶

run_streamlets()[source]¶

set_streamlet(index=None, location=None, source=None, destination=None)[source]¶

shuffle()[source]¶

max_stream_count = 5¶

System Call¶

sdss_access.sync.system_call.system_call(command, test=False, logger=None, logall=False, message=None, outname=None, errname=None)[source]¶

A convenient wrapper to log and perform system calls.

Parameters:

Parameters:	command (str) – The system command to run. It will be split internally by shlex.split(). test (bool, optional) – Set this to true to not actually run the commands. logger (logging.logging, optional) – If passed a logging object, diagnostic output will be set to this object. logall (bool, optional) – Set this to true to always log stdout and stderr (at level DEBUG). Otherwise stdout and stderr will only be logged for nonzero status message (str, optional) – Call logger.critical(message) with this message and then call sys.exit(status). outname (str, optional) – If set, use `outname` as the name of a file to contain stdout. It is the responsibility of the caller of this function to clean up the resulting file. Otherwise a temporary file will be used. errname (str, optional) – Same as `outname`, but will contain stderr.
Returns:	(status,out,err) (tuple) – The exit status, stdout and stderr of the process

command (str) – The system command to run. It will be split internally by shlex.split().
test (bool, optional) – Set this to true to not actually run the commands.
logger (logging.logging, optional) – If passed a logging object, diagnostic output will be set to this object.
logall (bool, optional) – Set this to true to always log stdout and stderr (at level DEBUG). Otherwise stdout and stderr will only be logged for nonzero status
message (str, optional) – Call logger.critical(message) with this message and then call sys.exit(status).
outname (str, optional) – If set, use outname as the name of a file to contain stdout. It is the responsibility of the caller of this function to clean up the resulting file. Otherwise a temporary file will be used.
errname (str, optional) – Same as outname, but will contain stderr.

Returns:

(status,out,err) (tuple) – The exit status, stdout and stderr of the process

Examples

>>> status,out,err = transfer.common.system_call('date')