sdss-access Reference

Path

exception sdss_access.path.path.AccessError

Bases: Exception

class sdss_access.path.path.BasePath(pathfile, mirror=False, public=False, release=None, verbose=False)

Bases: object

Class for construction of paths in general.

Variables:templates (dict) – The set of templates read from the configuration file.

any(filetype, **kwargs)

Checks if the local directory contains any of the type of file

Parameters:filetype (str) – File type parameter. Returns:any (bool) – Boolean indicating if the any files exist in the expanded path on disk.

dir(filetype, **kwargs)

Return the directory containing a file of a given type.

Parameters:filetype (str) – File type parameter. Returns:dir (str) – Directory containing the file.

exists(filetype, remote=None, **kwargs)

Checks if the given type of file exists locally

Parameters:

• filetype (str) – File type parameter.
• remote (bool) – If True, checks for remote existence of the file

Returns:

exists (bool) – Boolean indicating if the file exists.

expand(filetype, **kwargs)

Expand a wildcard path locally

Parameters:

• filetype (str) – File type parameter.
• as_url (bool) – Boolean to return SAS urls
• refine (str) – Regular expression string to filter the list of files by before random selection

Returns:

expand (list) – List of expanded full paths of the given type.

extract(name, example)

Extract keywords from an example path

full(filetype, **kwargs)

Return the full name of a given type of file.

Parameters:filetype (str) – File type parameter. Returns:full (str) – The full path to the file.

get_remote_base(scheme='https')

location(filetype, base_dir=None, **kwargs)

Return the location of the relative sas path of a given type of file.

Parameters:filetype (str) – File type parameter. Returns:full (str) – The relative sas path to the file.

lookup_keys(name)

Lookup the keyword arguments needed for a given path name

Parameters:name (str) – The name of the path Returns:A list of keywords needed for filepath generation

lookup_names()

Lookup what path names are available

Returns a list of the available path names in sdss_access. Use with lookup_keys to find the required keyword arguments for a given path name.

Returns:A list of the available path names.

name(filetype, **kwargs)

Return the directory containing a file of a given type.

Parameters:filetype (str) – File type parameter. Returns:name (str) – Name of a file with no directory information.

one(filetype, **kwargs)

Returns random one of the given type of file

Parameters:

• filetype (str) – File type parameter.
• as_url (bool) – Boolean to return SAS urls
• refine (str) – Regular expression string to filter the list of files by before random selection

Returns:

one (str) – Random file selected from the expanded list of full paths on disk.

random(filetype, **kwargs)

Returns random number of the given type of file

Parameters:

• filetype (str) – File type parameter.
• num (int) – The number of files to return
• as_url (bool) – Boolean to return SAS urls
• refine (str) – Regular expression string to filter the list of files by before random selection

Returns:

random (list) – Random file selected from the expanded list of full paths on disk.

refine(filelist, regex, filterdir='out', **kwargs)

Returns a list of files filterd by a regular expression

Parameters:

• filelist (list) – A list of files to filter on.
• regex (str) – The regular expression string to filter your list
• filterdir ({'in', 'out'}) – Indicates the filter to be inclusive or exclusive ‘out’ removes the items satisfying the regular expression ‘in’ keeps the items satisfying the regular expression

Returns:

refine (list) – A file list refined by an input regular expression.

replant_tree()

replants the tree based on release

set_base_dir(base_dir=None)

set_netloc(netloc=None, sdss=None, dtn=None)

set_remote_base(scheme=None)

url(filetype, base_dir=None, sasdir='sas', **kwargs)

Return the url of a given type of file.

Parameters:filetype (str) – File type parameter. Returns:full (str) – The sas url to the file.

class sdss_access.path.path.Path(mirror=False, public=False, release=None, verbose=False)

Bases: sdss_access.path.path.BasePath

Class for construction of paths in general. Sets a particular template file.

definitiondir(filetype, **kwargs)

Returns definition subdirectory in PLATELIST_DIR of the form: NNNNXX.

Parameters:

• filetype (str) – File type parameter.
• designid (int or str) – Design ID number. Will be converted to int internally.

Returns:

definitiondir (str) – Definition directory in the format NNNNXX.

platedir(filetype, **kwargs)

Returns plate subdirectory in PLATELIST_DIR of the form: NNNNXX/NNNNNN.

Parameters:

• filetype (str) – File type parameter.
• plateid (int or str) – Plate ID number. Will be converted to int internally.

Returns:

platedir (str) – Plate directory in the format NNNNXX/NNNNNN.

plateid6(filetype, **kwargs)

Print plate ID, accounting for 5-6 digit plate IDs.

Parameters:

• filetype (str) – File type parameter.
• plateid (int or str) – Plate ID number. Will be converted to int internally.

Returns:

plateid6 (str) – Plate ID formatted to a string of 6 characters.

spectrodir(filetype, **kwargs)

Returns SPECTRO_REDUX or BOSS_SPECTRO_REDUX depending on the value of run2d.

Parameters:

• filetype (str) – File type parameter.
• run2d (int or str) – 2D Reduction ID.

Returns:

spectrodir (str) – Value of the appropriate environment variable.

Sync

Access

class sdss_access.sync.access.Access(label='sdss_rsync', stream_count=5, mirror=False, public=False, release=None, verbose=False)

Bases: sdss_access.sync.rsync.RsyncAccess

Class for providing Rsync or Curl access depending on posix

Auth

class sdss_access.sync.auth.Auth(netloc=None, public=False, verbose=False)

Bases: object

load()

ready()

set_netloc(netloc=None)

set_netrc()

add the following username and password to your ~/.netrc file and remember to chmod 600 ~/.netrc

machine data.sdss.org login sdss password *-****

Windows: recommending _netrc following https://stackoverflow.com/questions/6031214/git-how-to-use-netrc-file-on-windows-to-save-user-and-password

set_password(password=None, inquire=False)

set_username(username=None, inquire=False)

Client

exception sdss_access.sync.cli.CliError

Bases: Exception

class sdss_access.sync.cli.Cli(label=None, data_dir=None, verbose=False)

Bases: object

Class for providing command line interface (cli) sync scripts, and logs to local disk

foreground_run(command, test=False, logger=None, logall=False, message=None, outname=None, errname=None)

A convenient wrapper to log and perform system calls.

Parameters:

• command (str) – The system command to run. It will be split internally by shlex.split().
• test (bool, optional) – Set this to true to not actually run the commands.
• logger (logging.logging, optional) – If passed a logging object, diagnostic output will be set to this object.
• logall (bool, optional) – Set this to true to always log stdout and stderr (at level DEBUG). Otherwise stdout and stderr will only be logged for nonzero status
• message (str, optional) – Call logger.critical(message) with this message and then call sys.exit(status).
• outname (str, optional) – If set, use outname as the name of a file to contain stdout. It is the responsibility of the caller of this function to clean up the resulting file. Otherwise a temporary file will be used.
• errname (str, optional) – Same as outname, but will contain stderr.

Returns:

(status,out,err) (tuple) – The exit status, stdout and stderr of the process

Examples

>>> status,out,err = transfer.common.system_call('date')

get_background_process(command=None, logfile=None, errfile=None, pause=1)

get_path(index=None)

set_dir()

wait_for_processes(processes, pause=60)

write_lines(path=None, lines=None)

tmp_dir = '/tmp'

Curl

class sdss_access.sync.curl.CurlAccess(label='sdss_curl', stream_count=5, mirror=False, public=False, release=None, verbose=False)

Bases: sdss_access.sync.baseaccess.BaseAccess

Class for providing Curl access to SDSS SAS Paths

check_file_exists_locally(destination=None, url_file_size=None, url_file_time=None)

Checks if file already exists (note that time check is only accurate to the minute)

generate_stream_task(task=None, out=None)

creates the task to put in the download stream

get_query_list(url_query)

Search through user specified “*” options and return all possible and valid url paths

get_task_status(task=None)

set_stream_task(task=None)

sets the path input dictionary for a task in a stream

set_url_list(query_path=None)

Gets url paths from get_query_list and returns file proparties and path

set_url_password(url_directory)

Authorize User on sas

access_mode = 'curl'

remote_scheme = 'https'

Http

class sdss_access.sync.http.HttpAccess(verbose=False, public=False, release=None, label='sdss_http')

Bases: sdss_access.path.path.Path

Class for providing HTTP access via urllib.request (python3) or urllib2 (python2) to SDSS SAS Paths

download_url_to_path(url, path, force=False)

Download a file from url via http, and put it at path

Parameters:

• url (str) – URL of file to download
• path (str) – local path to put file in

get(filetype, **kwargs)

Returns file name, downloading if remote access configured.

Parameters:

• filetype (str) – type of file
• arguments (keyword) – keywords to fully specify path

Notes

Path templates are defined in $DIMAGE_DIR/data/dimage_paths.ini

local()

Configures back to local access

remote(remote_base=None, username=None, password=None)

Configures remote access

Parameters:

• remote_base (str) – base URL path for remote repository
• username (str) – user name for remote repository
• password (str) – password for local repository

set_auth(username=None, password=None)

Rsync

class sdss_access.sync.rsync.RsyncAccess(label='sdss_rsync', stream_count=5, mirror=False, public=False, release=None, verbose=False)

Bases: sdss_access.sync.baseaccess.BaseAccess

Class for providing Rsync access to SDSS SAS Paths

generate_stream_task(task=None, out=None)

creates the task to put in the download stream

get_task_out(task=None)

set_stream_task(task=None)

sets the path input dictionary for a task in a stream

access_mode = 'rsync'

remote_scheme = 'rsync'

Stream

class sdss_access.sync.stream.Stream(stream_count=None, verbose=False)

Bases: object

append_streamlet(index=None, task=None)

append_task(location=None, source=None, destination=None)

append_tasks_to_streamlets(offset=None, limit=None)

commit_streamlet(streamlet=None)

commit_streamlets()

get_locations(offset=None, limit=None)

get_streamlet(index=None, increment=1)

refine_task(regex=None)

reset()

reset_streamlet()

reset_task()

run_streamlets()

set_streamlet(index=None, location=None, source=None, destination=None)

shuffle()

max_stream_count = 5

System Call

sdss_access.sync.system_call.system_call(command, test=False, logger=None, logall=False, message=None, outname=None, errname=None)

A convenient wrapper to log and perform system calls.

Parameters:

• command (str) – The system command to run. It will be split internally by shlex.split().
• test (bool, optional) – Set this to true to not actually run the commands.
• logger (logging.logging, optional) – If passed a logging object, diagnostic output will be set to this object.
• logall (bool, optional) – Set this to true to always log stdout and stderr (at level DEBUG). Otherwise stdout and stderr will only be logged for nonzero status
• message (str, optional) – Call logger.critical(message) with this message and then call sys.exit(status).
• outname (str, optional) – If set, use outname as the name of a file to contain stdout. It is the responsibility of the caller of this function to clean up the resulting file. Otherwise a temporary file will be used.
• errname (str, optional) – Same as outname, but will contain stderr.

Returns:

(status,out,err) (tuple) – The exit status, stdout and stderr of the process

Examples

>>> status,out,err = transfer.common.system_call('date')