elektronn2.utils package

Submodules

elektronn2.utils.cnncalculator module

elektronn2.utils.cnncalculator.cnncalculator(filters, poolings, desired_patch_size=None, mfp=False, force_center=False, desired_output=None, ndim=1)[source]

Helper to calculate CNN architectures

This is a function, but it returns an object that has various architecture values as attributes. Useful is also to simply print ‘d’ as in the example.

Parameters:
  • filters (list) – Filter shapes (for anisotropic filters the shapes are again a list)
  • poolings (list) – Pooling factors
  • desired_patch_size (int or list[int]) – Desired patch_size size(s). If None a range of suggestions can be found in the attribute valid_patch_sizes
  • mfp (list[bool] or bool) – Whether to apply Max-Fragment-Pooling in this Layer and check compliance with max-fragment-pooling (requires other patch_size sizes than normal pooling)
  • force_center (Bool) – Check if output neurons/pixel lie at center of patch_size neurons/pixel (and not in between)
  • desired_output (None or int or list[int]) – Alternative to desired_patch_size
  • ndim (int) – Dimensionality of CNN

Examples

Calculation for anisotropic “flat” 3d CNN with mfp in the first layers only:

>>> desired_patch_size   = [8, 211, 211]
>>> filters         = [[1,6,6], [4,4,4], [2,2,2], [1,1,1]]
>>> pool            = [[1,2,2], [2,2,2], [2,2,2], [1,1,1]]
>>> mfp             = [True,    True,    False,    False ]
>>> ndim=3
>>> d = cnncalculator(filters, pool, desired_patch_size, mfp=mfp, force_center=True, desired_output=None, ndim=ndim)
patch_size (8) changed to (10) (size too small)
patch_size (211) changed to (210) (size not possible)
patch_size (211) changed to (210) (size not possible)
>>> print(d)
patch_size: [10, 210, 210]
Layer/Fragment sizes:   [(1, 24, 24), (1, 24, 24), (3, 49, 49), (10, 102, 102)]
Unpooled Layer sizes:   [(1, 24, 24), (2, 48, 48), (7, 99, 99), (10, 205, 205)]
Receptive fields:       [(9, 23, 23), (9, 23, 23), (5, 15, 15), (1, 7, 7)]
Strides:                [(4, 8, 8), (4, 8, 8), (2, 4, 4), (1, 2, 2)]
Overlap:                [(5, 15, 15), (5, 15, 15), (3, 11, 11), (0, 5, 5)]
Offset:         [4.5, 11.5, 11.5]
If offset is non-int: output neurons lie centered on patch_size neurons,they have an odd FOV
elektronn2.utils.cnncalculator.get_cloesest_valid_patch_size(filters, poolings, desired_patch_size=100, mfp=False, ndim=1)[source]
elektronn2.utils.cnncalculator.get_valid_patch_sizes(filters, poolings, desired_patch_size=100, mfp=False, ndim=1)[source]

elektronn2.utils.gpu module

elektronn2.utils.gpu.get_free_gpu(wait=0, nb_gpus=-1, prefix='cuda')[source]
elektronn2.utils.gpu.initgpu(gpu, prefix='cuda')[source]

elektronn2.utils.legacy module

elektronn2.utils.legacy.create_cnn(config_file, n_ch, param_file=None, mfp=False, axis_order='theano', constant_weights=False, imposed_input_size=None)[source]
elektronn2.utils.legacy.load_params_into_model(param_file, model)[source]

Loads parameters directly from save file into a graph manager (this requires that the graph is identical to the cnn from the param file :param param_file: :param gm: :return:

elektronn2.utils.locking module

Implementation of a simple cross-platform file locking mechanism. This is a modified version of code retrieved on 2013-01-01 from http://www.evanfosmark.com/2009/01/cross-platform-file-locking-support-in-python. (The original code was released under the BSD License. See below for details.)

Modifications in this version:
  • Tweak docstrings for sphinx.
  • Accept an absolute path for the protected file (instead of a file name relative to cwd).
  • Allow timeout to be None.
  • Fixed a bug that caused the original code to be NON-threadsafe when the same FileLock instance was shared by multiple threads in one process. (The original was safe for multiple processes, but not multiple threads in a single process. This version is safe for both cases.)
  • Added purge() function.
  • Added available() function.
  • Expanded API to mimic threading.Lock interface: - __enter__ always calls acquire(), and therefore blocks if acquire() was called previously. - __exit__ always calls release(). It is therefore a bug to call release() from within a context manager. - Added locked() function. - Added blocking parameter to acquire() method

Warning

  • The locking mechanism used here may need to be changed to support old NFS filesystems: http://lwn.net/Articles/251004 (Newer versions of NFS should be okay, e.g. NFSv3 with Linux kernel 2.6. Check the open(2) man page for details about O_EXCL.)
  • This code has not been thoroughly tested on Windows, and there has been one report of incorrect results on Windows XP and Windows 7. The locking mechanism used in this class should (in theory) be cross-platform, but use at your own risk.

ORIGINAL LICENSE:

The original code did not properly include license text. (It merely said “License: BSD”.) Therefore, we’ll attach the following generic BSD License terms to this file. Those who extract this file from the lazyflow code base (LGPL) for their own use are therefore bound by the terms of both the Simplified BSD License below AND the LGPL.

Copyright (c) 2013, Evan Fosmark and others. All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

  1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
  2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

The views and conclusions contained in the software and documentation are those of the authors and should not be interpreted as representing official policies, either expressed or implied, of the FreeBSD Project.

class elektronn2.utils.locking.FileLock(protected_file_path, timeout=120, delay=1, lock_file_contents=None)[source]

Bases: object

A file locking mechanism that has context-manager support so you can use it in a with statement. This should be relatively cross compatible as it doesn’t rely on msvcrt or fcntl for the locking.

exception FileLockException[source]

Bases: exceptions.Exception

acquire(blocking=True)[source]

Acquire the lock, if possible. If the lock is in use, and blocking is False, return False. Otherwise, check again every self.delay seconds until it either gets the lock or exceeds timeout number of seconds, in which case it raises an exception.

available()[source]

Returns True iff the file is currently available to be locked.

locked()[source]

Returns True iff the file is owned by THIS FileLock instance. (Even if this returns false, the file could be owned by another FileLock instance, possibly in a different thread or process).

purge()[source]

For debug purposes only. Removes the lock file from the hard disk.

release()[source]

Get rid of the lock by deleting the lockfile. When working in a with statement, this gets automatically called at the end.

elektronn2.utils.plotting module

class elektronn2.utils.plotting.Scroller(axes, images, names, init_z=None)[source]

Bases: object

onscroll(event)[source]
update()[source]
elektronn2.utils.plotting.add_timeticks(ax, times, steps, time_str='mins', num=5)[source]
elektronn2.utils.plotting.embedfilters(filters, border_width=1, normalize=False, output_ratio=1.0, rgb_axis=None)[source]

Embed an nd array into an 2d matrix by tiling. The last two dimensions of a are assumed to be spatial, the others are tiled recursively.

elektronn2.utils.plotting.my_quiver(x, y, img=None, c=None)[source]

first dim of x,y changes along vertical axis second dim changes along horizontal axis x: vertical vector component y: horizontal vector component

elektronn2.utils.plotting.plot_debug(var, debug_output_names, save_name)[source]
elektronn2.utils.plotting.plot_exectimes(exectimes, save_path='~/exectimes.png', max_items=32)[source]

Plot model execution time dict obtained from elektronn2.neuromancer.model.Model.measure_exectimes()

Parameters:
  • exectimes – OrderedDict of execution times (output of Model.measure_exectimes())
  • save_path – Where to save the plot
  • max_items – Only the max_items largest execution times are given names and are plotted independently. Everything else is grouped under ‘(other nodes)’.
elektronn2.utils.plotting.plot_hist(timeline, history, save_name, loss_smoothing_length=200, autoscale=True)[source]

Plot graphical info during Training

elektronn2.utils.plotting.plot_kde(pred, target, save_name, limit=90, scale='same', grid=50, take_last=4000)[source]
elektronn2.utils.plotting.plot_regression(pred, target, save_name, loss_smoothing_length=200, autoscale=True)[source]

Plot graphical info during Training

elektronn2.utils.plotting.plot_trainingtarget(img, lab, stride=1)[source]

Plots raw image vs target to check if valid batches are produced. Raw data is also shown overlaid with targets

Parameters:
  • img (2d array) – raw image from batch
  • lab (2d array) – targets
  • stride (int) – strides of targets
elektronn2.utils.plotting.plot_var(var, save_name)[source]
elektronn2.utils.plotting.scroll_plot(images, names=None, init_z=None)[source]

Creates a plot 1x2 image plot of 3d volume images Scrolling changes the displayed slices

Parameters:
  • images (list of arrays (or single)) – Each array of shape (z,y,x) or (z,y,x,RGB)
  • names (list of strings (or single)) – Names for each image
  • Usage
  • -----
  • the scroll interaction to work, the "scroller" object (For) –
  • be returned to the calling scope (must) –
  • fig, scroller = _scroll_plot4(images, names) (>>>) –
  • fig.show() (>>>) –
elektronn2.utils.plotting.sma(c, n)[source]

Returns box-SMA of c with box length n, the returned array has the same length as c and is const-padded at the beginning

elektronn2.utils.ptk_completions module

Provides completions for the Python language, file sytem paths and a custom list of words
for the ELEKTRONN2/Elektronn prompt_toolkit shell.

This module is mostly based on - https://github.com/jonathanslenders/ptpython/blob/master/ptpython/completer.py - https://github.com/jonathanslenders/ptpython/blob/master/ptpython/utils.py (at git revision 32827385cca65eabefccb06b56e4cf9d2c1e0120), which both are available under the following license (thanks, Jonathan and contributors!):

Copyright (c) 2015, Jonathan Slenders All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

  • Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
  • Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
  • Neither the name of the {organization} nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

class elektronn2.utils.ptk_completions.NumaCompleter(get_globals, get_locals, words=None, words_metastring=u'')[source]

Bases: prompt_toolkit.completion.Completer

Completer for Python, file system paths and custom words

get_completions(document, complete_event)[source]

Get completions.

elektronn2.utils.utils_basic module

elektronn2.utils.utils_basic.get_free_cpu_count()[source]
elektronn2.utils.utils_basic.parallel_accum(func, n_ret, var_args, const_args, proc=-1, debug=False)[source]
class elektronn2.utils.utils_basic.timeit(*args, **kwargs)[source]

Bases: elektronn2.utils.utils_basic.DecoratorBase

class elektronn2.utils.utils_basic.cache(*args, **kwargs)[source]

Bases: elektronn2.utils.utils_basic.DecoratorBase

static hash_args(args)[source]
class elektronn2.utils.utils_basic.CircularBuffer(buffer_len)[source]

Bases: object

append(data)[source]
data
mean()[source]
setvals(val)[source]
class elektronn2.utils.utils_basic.AccumulationArray(right_shape=(), dtype=<type 'numpy.float32'>, n_init=100, data=None, ema_factor=0.95)[source]

Bases: object

add_offset(off)[source]
append(data)[source]
clear()[source]
data
ema
max()[source]
mean()[source]
min()[source]
sum()[source]
class elektronn2.utils.utils_basic.KDT(n_neighbors=5, radius=1.0, algorithm='auto', leaf_size=30, metric='minkowski', p=2, metric_params=None, n_jobs=1, **kwargs)[source]

Bases: sklearn.neighbors.unsupervised.NearestNeighbors

warning_shown = False
class elektronn2.utils.utils_basic.DynamicKDT(points=None, k=1, n_jobs=-1, rebuild_thresh=100, aniso_scale=None)[source]

Bases: object

append(point)[source]
get_knn(query_points, k=None)[source]
get_radius_nn(query_points, radius)[source]
elektronn2.utils.utils_basic.import_variable_from_file(file_path, class_name)[source]
elektronn2.utils.utils_basic.pickleload(file_name)[source]

Loads all object that are saved in the pickle file. Multiple objects are returned as list.

elektronn2.utils.utils_basic.picklesave(data, file_name)[source]

Writes one or many objects to pickle file

data:
single objects to save or iterable of objects to save. For iterable, all objects are written in this order to the file.
file_name: string
path/name of destination file
elektronn2.utils.utils_basic.h5save(data, file_name, keys=None, compress=True)[source]

Writes one or many arrays to h5 file

data:
single array to save or iterable of arrays to save. For iterable all arrays are written to the file.
file_name: string
path/name of destination file
keys: string / list thereof
For single arrays this is a single string which is used as a name for the data set. For multiple arrays each dataset is named by the corresponding key. If keys is None, the dataset names created by enumeration: data%i
compress: Bool
Whether to use lzf compression, defaults to True. Most useful for label arrays.
elektronn2.utils.utils_basic.h5load(file_name, keys=None)[source]

Loads data sets from h5 file

file_name: string
destination file
keys: string / list thereof
Load only data sets specified in keys and return as list in the order of keys For a single key the data is returned directly - not as list If keys is None all datasets that are listed in the keys-attribute of the h5 file are loaded.
elektronn2.utils.utils_basic.pretty_string_ops(n)[source]

Return a humanized string representation of a large number.

elektronn2.utils.utils_basic.pretty_string_time(t)[source]

Custom printing of elapsed time

elektronn2.utils.utils_basic.makeversiondir(path, dir_name=None, cd=False)[source]
class elektronn2.utils.utils_basic.Timer(silent_all=False)[source]

Bases: object

check(name=None, silent=False)[source]
plot(accum=False)[source]
summary(silent=False, print_func=None)[source]
elektronn2.utils.utils_basic.unique_rows(a)[source]
elektronn2.utils.utils_basic.as_list(var)[source]

Module contents