sdss-access Reference

Path

exception sdss_access.path.path.AccessError

Bases: Exception

class sdss_access.path.path.BasePath(release=None, public=False, mirror=False, verbose=False, force_modules=None, preserve_envvars=None)

Bases: object

Class for construction of paths in general.

Parameters:

• release (str) – The release name, e.g. ‘DR15’, ‘MPL-9’.

• public (bool) – If True, uses public urls. Only needed for public data releases. Automatically set to True when release contains “DR”.

• mirror (bool) – If True, uses the mirror data domain url. Default is False.

• verbose (bool) – If True, turns on verbosity. Default is False.

• force_modules (bool) – If True, forces svn or github software products to use any existing local Module environment paths, e.g. PLATEDESIGN_DIR

• preserve_envvars (bool | list) – Flag(s) to indicate some or all original environment variables to preserve

Variables:

templates (dict) – The set of templates read from the configuration file.

add_temp_path(name: str, path: str, envvar_path: str | None = None)

Add a temporary path template in sdss_access

Add a path template temporarily into the local os environment for use in sdss_access. Define a template name and path. The path must start with an environment variable definition.

This is useful for development of new paths before adding them to the tree and tagging a new version. This allows sdss_access to still be used in the interim. This is an alternative to checking out the tree git repo and modifying paths there. The recommended way of adding new paths is through a PR on the tree product.

Parameters:

• name (str) – the temporary file species name

• path (str) – the temporary template directory path

• envvar_path (str, optional) – the definition path of the environment variable, by default None

Raises:

• ValueError – when the name does not match the correct syntax

• ValueError – when the path does not start with an environment variable

• ValueError – when the environment variable is not defined

any(filetype, **kwargs)

Checks if the local directory contains any of the type of file

Parameters:

filetype (str) – File type parameter.

Returns:

any (bool) – Boolean indicating if the any files exist in the expanded path on disk.

static check_modules(template, permanent=None)

Check for any existing Module path environment

For software product paths, overrides the tree environment paths with existing original envvars from os.environ that may be set from shell bash or module environments. Checks the original os.environ for any environment variables and replaces the template envvar with the original os version. Ignores all SAS data paths. If permanent is True, then permanently replaces the envvar in existing os.environ with the original. Assumes original environment variables points to definitions created by module files or bash profiles.

Parameters:

• template (str) – The path template to check

• permanent (bool) – If True, sets the original module environment variable into os.environ

Returns:

The template with updated environment variable path

dir(filetype, **kwargs)

Return the directory containing a file of a given type.

Parameters:

filetype (str) – File type parameter.

Returns:

dir (str) – Directory containing the file.

exists(filetype, remote=None, **kwargs)

Checks if the given type of file exists locally

Parameters:

• filetype (str) – File type parameter.

• remote (bool) – If True, checks for remote existence of the file

Returns:

exists (bool) – Boolean indicating if the file exists.

expand(filetype, **kwargs)

Expand a wildcard path locally

Parameters:

• filetype (str) – File type parameter.

• as_url (bool) – Boolean to return SAS urls

• refine (str) – Regular expression string to filter the list of files by before random selection

Returns:

expand (list) – List of expanded full paths of the given type.

extract(name, example)

Extract keywords from an example path

Attempts to extract the defined keyword values from an example filepath for a given path name. The filepath must be a full SDSS SAS filepath.

Parameters:

• name (str) – The name of the path definition

• example (str) – The absolute filepath to the example file

Returns:

A dictionary of path keyword values

Example

>>> from sdss_access.path import Path
>>> path = Path()
>>> filepath = '/Users/Brian/Work/sdss/sas/mangawork/manga/spectro/redux/v2_5_3/8485/stack/manga-8485-1901-LOGCUBE.fits'
>>> path.extract('mangacube', filepath)
>>> {'drpver': 'v2_5_3', 'plate': '8485', 'ifu': '1901', 'wave': 'LOG'}

find_location(filetype, **kwargs)

Finds a relative location of a product path

Attempts to find a relative path location for a software product path. Loops over all product_roots defined in the tree and tests if a relative location can be extracted, i.e. if the path starts with a given root path. The root environment paths searched are the following in order of precendence: PRODUCT_ROOT, SDSS_SVN_ROOT, SDSS_INSTALL_PRODUCT_ROOT, SDSS_PRODUCT_ROOT, SDSS4_PRODUCT_ROOT. If no root is found uses one directory up from SAS_BASE_DIR.

Parameters:

• filetype (str) – File type parameter.

• kwargs (dict) – Path definition keyword arguments

Returns:

The relative path location (to the base_dir)

full(filetype, **kwargs)

Return the full local path of a given type of file.

Parameters:

• filetype (str) – File type parameter.

• force_module (bool) – If True, forces software products to use any existing Module environment paths

• kwargs (dict) – Any path template keyword arguments

Returns:

full (str) – The full local path to the file.

static get_available_releases(public=None)

Get the available releases

Parameters:

public (bool) – If True, only return public data releases

get_netloc(netloc=None, sdss=None, sdss5=None, dtn=None, svn=None, mirror=None)

Get a net url domain

Returns an SDSS url domain location. Options are the SDSS SAS domain, the rsync download server, the svn server, or the mirror data domain. The mirror data domain is retrieved either by the mirror input keyword argument or by the path.mirror attribute.

Parameters:

• netloc (str) – An exact net location to return directly

• sdss (bool) – If True, returns SDSS data domain: data.sdss.org

• sdss5 (bool) – If True, sets the SDSS-V data domain: data.sdss5.org

• dtn (bool) – If True, returns SDSS rsync server domain: dtn.sdss.org

• svn (bool) – If True, returns SDSS svn domain: svn.sdss.org

• mirror (bool) – If True, return SDSS mirror domain: data.mirror.sdss.org.

Returns:

An http domain name

get_remote_base(scheme='https', svn=None)

Get the remote base path

Parameters:

• scheme (str) – The url scheme. Either “https” or “rsync”.

• svn (bool) – If True, uses the svn url domain svn.sdss.org as the netloc

has_name(name)

Check if a given path name exists in the set of templates

Parameters:

name (str) – The path name to lookup

is_sdss5() → bool

Checks if the release is an SDSS-V work or ipl release

location(filetype, base_dir=None, **kwargs)

Return the location of the relative sas path of a given type of file.

Parameters:

• filetype (str) – File type parameter.

• base_dir (str) – A root directory to use as the base. Defaults to SAS_BASE_DIR.

Returns:

The relative path location (to the base_dir)

lookup_keys(name)

Lookup the keyword arguments needed for a given path name

Parameters:

name (str) – The name of the path

Returns:

A list of keywords needed for filepath generation

lookup_names()

Lookup what path names are available

Returns a list of the available path names in sdss_access. Use with lookup_keys to find the required keyword arguments for a given path name.

Returns:

A list of the available path names.

name(filetype, **kwargs)

Return the name of a file of a given type.

Parameters:

filetype (str) – File type parameter.

Returns:

name (str) – Name of a file with no directory information.

one(filetype, **kwargs)

Returns random one of the given type of file

Parameters:

• filetype (str) – File type parameter.

• as_url (bool) – Boolean to return SAS urls

• refine (str) – Regular expression string to filter the list of files by before random selection

Returns:

one (str) – Random file selected from the expanded list of full paths on disk.

random(filetype, **kwargs)

Returns random number of the given type of file

Parameters:

• filetype (str) – File type parameter.

• num (int) – The number of files to return

• as_url (bool) – Boolean to return SAS urls

• refine (str) – Regular expression string to filter the list of files by before random selection

Returns:

random (list) – Random file selected from the expanded list of full paths on disk.

refine(filelist, regex, filterdir='out', **kwargs)

Returns a list of files filterd by a regular expression

Parameters:

• filelist (list) – A list of files to filter on.

• regex (str) – The regular expression string to filter your list

• filterdir ({'in', 'out'}) – Indicates the filter to be inclusive or exclusive ‘out’ removes the items satisfying the regular expression ‘in’ keeps the items satisfying the regular expression

Returns:

refine (list) – A file list refined by an input regular expression.

replant_tree(release=None)

Replants the tree based on release

Resets the path definitions given a specified release

Parameters:

release (str) – A release to use when replanting the tree

set_base_dir(base_dir=None)

Sets the base directory

Sets the base_dir attribute. Defaults to $SAS_BASE_DIR. Can be overridden by passing in base_dir keyword argument. The base_dir sets the beginning part of all local paths.

Parameters:

base_dir (str) – A directory path to use as the base

set_netloc(netloc=None, sdss=None, sdss5=None, dtn=None, svn=None, mirror=None)

Set a url domain location

Sets an SDSS url domain location. Options are the SDSS SAS domain, the rsync download server, the svn server, or the mirror data domain. The mirror data domain is set either by the mirror input keyword argument or by the path.mirror attribute.

Parameters:

• netloc (str) – An exact net location to use directly

• sdss (bool) – If True, sets the SDSS-IV data domain: data.sdss.org

• sdss5 (bool) – If True, sets the SDSS-V data domain: data.sdss5.org

• dtn (bool) – If True, sets the SDSS rsync server domain: dtn.sdss.org

• svn (bool) – If True, sets the SDSS svn domain: svn.sdss.org

• mirror (bool) – If True, sets the SDSS mirror domain: data.mirror.sdss.org.

set_remote_base(scheme='https')

url(filetype, base_dir=None, sasdir='sas', **kwargs)

Return the url of a given type of file.

Parameters:

• filetype (str) – File type parameter.

• base_dir (str) – A root directory to use as the base. Defaults to SAS_BASE_DIR.

Returns:

full (str) – The sas url to the file.

static yield_product_root()

yields a product root environment name

class sdss_access.path.path.Path(release=None, public=False, mirror=False, verbose=False, force_modules=None, preserve_envvars=None)

Bases: BasePath

Class for construction of paths in general. Sets a particular template file.

Parameters:

• release (str) – The release name, e.g. ‘DR15’, ‘MPL-9’.

• public (bool) – If True, uses public urls. Only needed for public data releases. Automatically set to True when release contains “DR”.

• mirror (bool) – If True, uses the mirror data domain url. Default is False.

• verbose (bool) – If True, turns on verbosity. Default is False.

• force_modules (bool) – If True, forces svn or github software products to use any existing local Module environment paths, e.g. PLATEDESIGN_DIR

• preserve_envvars (bool | list) – Flag(s) to indicate some or all original environment variables to preserve

Variables:

templates (dict) – The set of templates read from the configuration file.

apginst(filetype, **kwargs)

Returns APOGEE “instrument” from “telescope”.

Parameters:

• filetype (str) – File type parameter.

• telescope (str) – The APOGEE telescope (apo25m, lco25m, apo1m).

Returns:

instrument (str) – The APOGEE instrument (apogee-n, apogee-s).

apgprefix(filetype, **kwargs)

Returns APOGEE prefix using telescope/instrument.

Parameters:

• filetype (str) – File type parameter.

• telescope (str) – The APOGEE telescope (apo25m, lco25m, apo1m).

• instrument (str) – The APOGEE instrument (apogee-n, apogee-s).

Returns:

prefix (str) – The APOGEE prefix (ap/as).

cat_id_groups(filetype, **kwargs)

Return a folder structure to group data together based on their catalog identifier so that we don’t have too many files in any one folder.

Parameters:

• filetype (str) – File type parameter.

• cat_id (int or str) – SDSS-V catalog identifier

Returns:

catalogid_group (str) – A set of folders.

component_default(filetype, **kwargs)

Return the component name, if given.

The component designates a stellar or planetary body following the Washington Multiplicity Catalog, which was adopted by the XXIV meeting of the International Astronomical Union. When no component is given, the star is assumed to be without a discernible companion. When a component is given it follows the system (Hessman et al., arXiv:1012.0707):

– the brightest component is called “A”, whether it is initially resolved

into sub-components or not;

– subsequent distinct components not contained within “A” are labeled “B”,

“C”, etc.;

– sub-components are designated by the concatenation of on or more suffixes

with the primary label, starting with lowercase letters for the 2nd hierarchical level and then with numbers for the 3rd.

Parameters:

• filetype (str) – File type parameter. This argument is not used here, but is required for all special functions in the sdss_access product.

• component (str [optional]) – The component name as given by the fields.

Returns:

component (str) – The component name if given, otherwise a blank string.

configgrp(filetype, **kwargs)

Returns configuration summary file group subdirectory

Parameters:

• filetype (str) – File type parameter.

• configid (int or str) – Configuration ID number. Will be converted to int internally.

Returns:

configgrp (str) – Configuration group directory in the format NNNNXX.

configsubmodule(filetype, **kwargs)

Returns configuration summary submodule group subdirectory

Parameters:

• filetype (str) – File type parameter.

• configid (int or str) – Configuration ID number. Will be converted to int internally.

Returns:

configsubmodule (str) – Configuration submodule directory in the format NNNXXX.

definitiondir(filetype, **kwargs)

Returns definition subdirectory in PLATELIST_DIR of the form: NNNNXX.

Parameters:

• filetype (str) – File type parameter.

• designid (int or str) – Design ID number. Will be converted to int internally.

Returns:

definitiondir (str) – Definition directory in the format NNNNXX.

epochflag(filetype, **kwargs)

Returns the flag for epoch coadds for the BOSS idlspec2d

Parameters:

• filetype (str) – File type parameter

• run2d (str) – BOSS idlspec2d run2d version

Returns:

epochflag (str)

fieldgrp(filetype, **kwargs)

Returns the fieldid group for the BOSS idlspec2d run2d version

Parameters:

• filetype (str) – File type parameter

• run2d (str) – BOSS idlspec2d run2d version

• fieldid (str or int) – Field ID number. Will be converted to str internally.

Returns:

fieldgrp (str)

healpixgrp(filetype, **kwargs)

Returns HEALPIX group subdirectory

Parameters:

• filetype (str) – File type parameter.

• healpix (int or str) – HEALPix number. Will be converted to int internally.

Returns:

healpixgrp (str) – HEALPix group directory, HEALPix//1000.

isplate(filetype, **kwargs)

Returns the plate flag for BOSS idlspec2d run2d versions that utilize it

Parameters:

• filetype (str) – File type parameter

• run2d (str) – BOSS idlspec2d run2d version

Returns:

isplate (str) – isplate flag = ‘p’ for relevent run2d plates else flag = ‘’

pad_fieldid(filetype, **kwargs)

Returns the fieldid zero padded to its proper length for the BOSS idlspec2d run2d version

Parameters:

• filetype (str) – File type parameter

• run2d (str) – BOSS idlspec2d run2d version

• fieldid (str or int) – Field ID number. Will be converted to str internally.

Returns:

fieldid (str) – padd_fieldid in the form of N*’0’ where N is the number of necessary zeros to pad fieldid

platedir(filetype, **kwargs)

Returns plate subdirectory in PLATELIST_DIR of the form: NNNNXX/NNNNNN.

Parameters:

• filetype (str) – File type parameter.

• plateid (int or str) – Plate ID number. Will be converted to int internally.

Returns:

platedir (str) – Plate directory in the format NNNNXX/NNNNNN.

plategrp(filetype, **kwargs)

Returns plate group subdirectory

Parameters:

• filetype (str) – File type parameter.

• plate (int or str) – Plate ID number. Will be converted to int internally.

Returns:

plategrp (str) – Plate group directory in the format NNNNXX.

plateid6(filetype, **kwargs)

Print plate ID, accounting for 5-6 digit plate IDs.

Parameters:

• filetype (str) – File type parameter.

• plateid (int or str) – Plate ID number. Will be converted to int internally.

Returns:

plateid6 (str) – Plate ID formatted to a string of 6 characters.

sdss_id_groups(filetype, **kwargs)

Return a folder structure to group data together based on their SDSS identifier so that we don’t have too many files in any one folder.

Parameters:

• filetype (str) – File type parameter.

• sdss_id (int or str) – SDSS-V identifier

Returns:

sdssid_groups (str) – A set of folders.

spcoaddfolder(filetype, **kwargs)

Returns the reorganized subfolder structure for the BOSS idlspec2d run2d version

Parameters:

• filetype (str) – File type parameter

• run2d (str) – BOSS idlspec2d run2d version

• coadd (str) – Name of the custom coadd schema

Returns:

sptypefolder (str)

spcoaddgrp(filetype, **kwargs)

Returns the coadd group (field group analog) subfolder structure for the BOSS idlspec2d run2d version

Parameters:

• filetype (str) – File type parameter

• run2d (str) – BOSS idlspec2d run2d version

• coadd (str) – Name of the custom coadd schema

Returns:

spcoaddgrp (str)

spcoaddobs(filetype, **kwargs)

Returns the formatted observatory flag for custom coadds for the BOSS idlspec2d

Parameters:

• filetype (str) – File type parameter

• run2d (str) – BOSS idlspec2d run2d version

• obs (str) – Observatory of observations: LCO, APO, ‘’(for merged)

Returns:

obs (str)

spectrodir(filetype, **kwargs)

Returns SPECTRO_REDUX or BOSS_SPECTRO_REDUX depending on the value of run2d.

Parameters:

• filetype (str) – File type parameter.

• run2d (int or str) – 2D Reduction ID.

Returns:

spectrodir (str) – Value of the appropriate environment variable.

sptypefolder(filetype, **kwargs)

Returns the reorganized subfolder structure for the BOSS idlspec2d run2d version

Parameters:

• filetype (str) – File type parameter

• run2d (str) – BOSS idlspec2d run2d version

Returns:

sptypefolder (str)

tilegrp(filetype, **kwargs)

Returns LVM tile id group subdirectory

Parameters:

• filetype (str) – File type parameter.

• tileid (int or str) – LVM Tile ID number. Will be converted to int internally.

Returns:

tileidgrp (str) – Tile ID group directory in the format NNNNXX.

sdss_access.path.path.check_public_release(release: str | None = None, public: bool = False) → bool

Check if a release is public

Checks a given release to see if it is public. A release is public if it contains “DR” in the release name, and if todays date is <= the release_date as specified in the Tree.

Parameters:

• release (str) – The name of the release to check

• public (bool) – If True, force the release to be public

Returns:

bool – If the release if public

Raises:

AttributeError – when tree does not have a valid release date for a DR tree config

Sync

Access

class sdss_access.sync.access.Access(label='sdss_rsync', stream_count=5, mirror=False, public=False, release=None, verbose=False)

Bases: RsyncAccess

Class for providing Rsync or Curl access depending on posix

access_mode = 'rsync'

Auth

class sdss_access.sync.auth.Auth(netloc=None, public=False, verbose=False)

Bases: object

class for setting up SAS authenticaton for SDSS users

load()

Sets the username and password from the local netrc file

ready()

set_netloc(netloc=None)

sets the url domain location

set_netrc()

add the following username and password to your ~/.netrc file and remember to chmod 600 ~/.netrc

For SDSS-IV access: machine data.sdss.org login sdss password *-****

For SDSS-V access: machine data.sdss5.org login sdss5 password ***-*

Windows: recommending _netrc following https://stackoverflow.com/questions/6031214/git-how-to-use-netrc-file-on-windows-to-save-user-and-password

set_password(password=None, inquire=None)

sets the authentication password

Parameters:

• password – str The password for SDSS SAS access

• inquire – bool If True, prompts for input of password.

set_username(username=None, inquire=None)

sets the authentication username

Parameters:

• username – str The username for SDSS SAS access

• inquire – bool If True, prompts for input of username.

class sdss_access.sync.auth.AuthMixin

Bases: object

Mixin class to provide authentication method to other classes

set_auth(username=None, password=None, inquire=True)

Set the authentication

Parameters:

• username – str The username for SDSS SAS access

• password – str The password for SDSS SAS access

• inquire – bool If True, prompts for input of username/password.

BaseAccess

class sdss_access.sync.baseaccess.BaseAccess(label=None, stream_count=5, mirror=False, public=False, release=None, verbose=False, force_modules=None, preserve_envvars=None)

Bases: AuthMixin, Path

Class for providing Rsync or Curl access to SDSS SAS Paths

add(filetype, **kwargs)

Adds a filepath into the list of tasks to download

add_file(path, input_type=None)

Adds a file into the list of tasks to download

Adds a full filepath, url, or location to the list of tasks to download. This takes advantage of sdss_access parallel streams to download a list of files.

This is similar to the .add method, except this takes full filepaths or urls as input, whereas the .add method is better when inputing a product name and path template kwargs.

Parameters:

• path (str) – the filepath, url, or location

• input_type (str, optional) – the type of input, by default None

commit(offset=None, limit=None, follow_symlinks: bool = True)

Start the download

abstract generate_stream_task(task=None, out=None)

creates the task to put in the download stream

get_locations(offset=None, limit=None)

Rreturn the locations for all paths in the stream

get_paths(offset=None, limit=None)

Return the base paths for all paths in the stream

get_stream()

return a Stream object

get_urls(offset=None, limit=None)

Return the urls for all paths in the stream

remote(username=None, password=None, inquire=None)

Configures remote access

reset()

Reset all streams

set_stream()

Sets the download streams

set_stream_task(task=None, out=None)

sets the path input dictionary for a task in a stream

shuffle()

Shuffle the stream

access_mode = 'rsync'

remote_scheme = None

Client

exception sdss_access.sync.cli.CliError

Bases: Exception

class sdss_access.sync.cli.Cli(label=None, data_dir=None, verbose=False)

Bases: object

Class for providing command line interface (cli) sync scripts, and logs to local disk

foreground_run(command, test=False, logger=None, logall=False, message=None, outname=None, errname=None)

A convenient wrapper to log and perform system calls.

Parameters:

• command (str) – The system command to run. It will be split internally by shlex.split().

• test (bool, optional) – Set this to true to not actually run the commands.

• logger (logging.logging, optional) – If passed a logging object, diagnostic output will be set to this object.

• logall (bool, optional) – Set this to true to always log stdout and stderr (at level DEBUG). Otherwise stdout and stderr will only be logged for nonzero status

• message (str, optional) – Call logger.critical(message) with this message and then call sys.exit(status).

• outname (str, optional) – If set, use outname as the name of a file to contain stdout. It is the responsibility of the caller of this function to clean up the resulting file. Otherwise a temporary file will be used.

• errname (str, optional) – Same as outname, but will contain stderr.

Returns:

(status,out,err) (tuple) – The exit status, stdout and stderr of the process

Examples

>>> status,out,err = transfer.common.system_call('date')

get_background_process(command=None, logfile=None, errfile=None, pause=1)

get_path(index=None)

set_dir()

wait_for_processes(processes, pause=5, n_tasks=None, tasks_per_stream=None)

write_lines(path=None, lines=None)

tmp_dir = '/tmp'

tmp_exists = True

Curl

class sdss_access.sync.curl.CurlAccess(label='sdss_curl', stream_count=5, mirror=False, public=False, release=None, verbose=False)

Bases: BaseAccess

Class for providing Curl access to SDSS SAS Paths

check_file_exists_locally(destination=None, url_file_size=None, url_file_time=None)

Checks if file already exists (note that time check is only accurate to the minute)

generate_stream_task(task=None, out=None)

creates the task to put in the download stream

get_query_list(url_query)

Search through user specified “*” options and return all possible and valid url paths

get_task_status(task=None)

set_stream_task(task=None)

sets the path input dictionary for a task in a stream

set_url_list(query_path=None)

Gets url paths from get_query_list and returns file proparties and path

set_url_password(url_directory)

Authorize User on sas

access_mode = 'curl'

remote_scheme = 'https'

Http

class sdss_access.sync.http.HttpAccess(verbose=None, public=None, release=None, label='sdss_http')

Bases: AuthMixin, Path

Class for providing HTTP access via urllib.request (python3) or urllib2 (python2) to SDSS SAS Paths

download_url_to_path(url, path, force=False)

Download a file from url via http, and put it at path

Parameters:

• url (str) – URL of file to download

• path (str) – local path to put file in

get(filetype, **kwargs)

Returns file name, downloading if remote access configured.

Parameters:

• filetype (str) – type of file

• arguments (keyword) – keywords to fully specify path

Notes

Path templates are defined in $DIMAGE_DIR/data/dimage_paths.ini

local()

Configures back to local access

remote(remote_base=None, username=None, password=None)

Configures remote access

Parameters:

• remote_base (str) – base URL path for remote repository

• username (str) – user name for remote repository

• password (str) – password for local repository

Rsync

class sdss_access.sync.rsync.RsyncAccess(label='sdss_rsync', stream_count=5, mirror=False, public=False, release=None, verbose=False)

Bases: BaseAccess

Class for providing Rsync access to SDSS SAS Paths

generate_stream_task(task=None, out=None)

creates the task to put in the download stream

get_task_out(task=None)

set_stream_task(task=None)

sets the path input dictionary for a task in a stream

access_mode = 'rsync'

remote_scheme = 'rsync'

Stream

class sdss_access.sync.stream.Stream(stream_count=None, verbose=False)

Bases: object

append_streamlet(index=None, task=None)

append_task(sas_module=None, location=None, source=None, destination=None)

append_tasks_to_streamlets(offset=None, limit=None)

commit_streamlet(streamlet=None)

commit_streamlets()

get_locations(offset=None, limit=None)

get_streamlet(index=None, increment=1)

refine_task(regex=None)

reset()

reset_streamlet()

reset_task()

run_streamlets()

set_streamlet(index=None, sas_module=None, location=None, source=None, destination=None)

shuffle()

max_stream_count = 5

System Call

sdss_access.sync.system_call.system_call(command, test=False, logger=None, logall=False, message=None, outname=None, errname=None)

A convenient wrapper to log and perform system calls.

Parameters:

• command (str) – The system command to run. It will be split internally by shlex.split().

• test (bool, optional) – Set this to true to not actually run the commands.

• logger (logging.logging, optional) – If passed a logging object, diagnostic output will be set to this object.

• logall (bool, optional) – Set this to true to always log stdout and stderr (at level DEBUG). Otherwise stdout and stderr will only be logged for nonzero status

• message (str, optional) – Call logger.critical(message) with this message and then call sys.exit(status).

• outname (str, optional) – If set, use outname as the name of a file to contain stdout. It is the responsibility of the caller of this function to clean up the resulting file. Otherwise a temporary file will be used.

• errname (str, optional) – Same as outname, but will contain stderr.

Returns:

(status,out,err) (tuple) – The exit status, stdout and stderr of the process

Examples

>>> status,out,err = transfer.common.system_call('date')

Changelog Utilities

sdss_access.path.changelog.compute_changelog(new, old, pprint=None, to_list=None)

Compute the difference between two Tree PATH sections

Compares two tree PATH ini sections from the given environment configurations and returns adictionary with keys new, and updated, indicating newly added paths, and any paths that have been modified from the last release. Accepts either string names of config files, e.g. “dr16” and “dr15”, or the preloaded Tree configs, e.g. Tree(config='dr16').

Parameters:

• new (str|Tree) – The new tree enviroment to compare

• old (str|Tree) – The old tree environment to compare

• pprint (bool) – If True, returns a single joined string for printing. Default is False.

• to_list (bool) – If True, returns a list of strings formatted for printing. Default is False.

Returns:

A dictionary of relevant changes between the two releases

sdss_access.path.changelog.get_available_releases(public=None)

Get the available releases

Parameters:

public (bool) – If True, only return public data releases

sdss_access.path.changelog.get_path_templates(name, public=None)

Return the path templates for all releases

Produces a dictionary of path templates for a given path name for all releases. Set public keyword to toggle only public data releases (DRs).

Parameters:

• name (str) – The sdss_access path name

• public (bool) – If True, selects only public data releases

Returns:

A dictionary of the path template per release