elektronn2.utils package¶
Subpackages¶
Submodules¶
elektronn2.utils.cnncalculator module¶
-
elektronn2.utils.cnncalculator.
cnncalculator
(filters, poolings, desired_patch_size=None, mfp=False, force_center=False, desired_output=None, ndim=1)[source]¶ Helper to calculate CNN architectures
This is a function, but it returns an object that has various architecture values as attributes. Useful is also to simply print ‘d’ as in the example.
Parameters: - filters (list) – Filter shapes (for anisotropic filters the shapes are again a list)
- poolings (list) – Pooling factors
- desired_patch_size (int or list[int]) – Desired patch_size size(s). If
None
a range of suggestions can be found in the attributevalid_patch_sizes
- mfp (list[bool] or bool) – Whether to apply Max-Fragment-Pooling in this Layer and check compliance with max-fragment-pooling (requires other patch_size sizes than normal pooling)
- force_center (Bool) – Check if output neurons/pixel lie at center of patch_size neurons/pixel (and not in between)
- desired_output (None or int or list[int]) – Alternative to
desired_patch_size
- ndim (int) – Dimensionality of CNN
Examples
Calculation for anisotropic “flat” 3d CNN with mfp in the first layers only:
>>> desired_patch_size = [8, 211, 211] >>> filters = [[1,6,6], [4,4,4], [2,2,2], [1,1,1]] >>> pool = [[1,2,2], [2,2,2], [2,2,2], [1,1,1]] >>> mfp = [True, True, False, False ] >>> ndim=3 >>> d = cnncalculator(filters, pool, desired_patch_size, mfp=mfp, force_center=True, desired_output=None, ndim=ndim) patch_size (8) changed to (10) (size too small) patch_size (211) changed to (210) (size not possible) patch_size (211) changed to (210) (size not possible) >>> print(d) patch_size: [10, 210, 210] Layer/Fragment sizes: [(1, 24, 24), (1, 24, 24), (3, 49, 49), (10, 102, 102)] Unpooled Layer sizes: [(1, 24, 24), (2, 48, 48), (7, 99, 99), (10, 205, 205)] Receptive fields: [(9, 23, 23), (9, 23, 23), (5, 15, 15), (1, 7, 7)] Strides: [(4, 8, 8), (4, 8, 8), (2, 4, 4), (1, 2, 2)] Overlap: [(5, 15, 15), (5, 15, 15), (3, 11, 11), (0, 5, 5)] Offset: [4.5, 11.5, 11.5] If offset is non-int: output neurons lie centered on patch_size neurons,they have an odd FOV
elektronn2.utils.gpu module¶
elektronn2.utils.legacy module¶
elektronn2.utils.locking module¶
Implementation of a simple cross-platform file locking mechanism. This is a modified version of code retrieved on 2013-01-01 from http://www.evanfosmark.com/2009/01/cross-platform-file-locking-support-in-python. (The original code was released under the BSD License. See below for details.)
- Modifications in this version:
- Tweak docstrings for sphinx.
- Accept an absolute path for the protected file (instead of a file name relative to cwd).
- Allow timeout to be None.
- Fixed a bug that caused the original code to be NON-threadsafe when the same FileLock instance was shared by multiple threads in one process. (The original was safe for multiple processes, but not multiple threads in a single process. This version is safe for both cases.)
- Added
purge()
function. - Added
available()
function. - Expanded API to mimic
threading.Lock interface
: -__enter__
always callsacquire()
, and therefore blocks ifacquire()
was called previously. -__exit__
always callsrelease()
. It is therefore a bug to callrelease()
from within a context manager. - Addedlocked()
function. - Added blocking parameter toacquire()
method
Warning
- The locking mechanism used here may need to be changed to support old NFS filesystems: http://lwn.net/Articles/251004 (Newer versions of NFS should be okay, e.g. NFSv3 with Linux kernel 2.6. Check the open(2) man page for details about O_EXCL.)
- This code has not been thoroughly tested on Windows, and there has been one report of incorrect results on Windows XP and Windows 7. The locking mechanism used in this class should (in theory) be cross-platform, but use at your own risk.
ORIGINAL LICENSE:
The original code did not properly include license text. (It merely said “License: BSD”.) Therefore, we’ll attach the following generic BSD License terms to this file. Those who extract this file from the lazyflow code base (LGPL) for their own use are therefore bound by the terms of both the Simplified BSD License below AND the LGPL.
Copyright (c) 2013, Evan Fosmark and others. All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
- Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
- Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
The views and conclusions contained in the software and documentation are those of the authors and should not be interpreted as representing official policies, either expressed or implied, of the FreeBSD Project.
-
class
elektronn2.utils.locking.
FileLock
(protected_file_path, timeout=120, delay=1, lock_file_contents=None)[source]¶ Bases:
object
A file locking mechanism that has context-manager support so you can use it in a
with
statement. This should be relatively cross compatible as it doesn’t rely onmsvcrt
orfcntl
for the locking.-
acquire
(blocking=True)[source]¶ Acquire the lock, if possible. If the lock is in use, and blocking is False, return False. Otherwise, check again every self.delay seconds until it either gets the lock or exceeds timeout number of seconds, in which case it raises an exception.
-
elektronn2.utils.plotting module¶
-
elektronn2.utils.plotting.
embedfilters
(filters, border_width=1, normalize=False, output_ratio=1.0, rgb_axis=None)[source]¶ Embed an nd array into an 2d matrix by tiling. The last two dimensions of
a
are assumed to be spatial, the others are tiled recursively.
-
elektronn2.utils.plotting.
my_quiver
(x, y, img=None, c=None)[source]¶ first dim of x,y changes along vertical axis second dim changes along horizontal axis x: vertical vector component y: horizontal vector component
-
elektronn2.utils.plotting.
plot_exectimes
(exectimes, save_path='~/exectimes.png', max_items=32)[source]¶ Plot model execution time dict obtained from elektronn2.neuromancer.model.Model.measure_exectimes()
Parameters: - exectimes – OrderedDict of execution times (output of Model.measure_exectimes())
- save_path – Where to save the plot
- max_items – Only the max_items largest execution times are given names and are plotted independently. Everything else is grouped under ‘(other nodes)’.
-
elektronn2.utils.plotting.
plot_hist
(timeline, history, save_name, loss_smoothing_length=200, autoscale=True)[source]¶ Plot graphical info during Training
-
elektronn2.utils.plotting.
plot_kde
(pred, target, save_name, limit=90, scale='same', grid=50, take_last=4000)[source]¶
-
elektronn2.utils.plotting.
plot_regression
(pred, target, save_name, loss_smoothing_length=200, autoscale=True)[source]¶ Plot graphical info during Training
-
elektronn2.utils.plotting.
plot_trainingtarget
(img, lab, stride=1)[source]¶ Plots raw image vs target to check if valid batches are produced. Raw data is also shown overlaid with targets
Parameters: - img (2d array) – raw image from batch
- lab (2d array) – targets
- stride (int) – strides of targets
-
elektronn2.utils.plotting.
scroll_plot
(images, names=None, init_z=None)[source]¶ Creates a plot 1x2 image plot of 3d volume images Scrolling changes the displayed slices
Parameters: - images (list of arrays (or single)) – Each array of shape (z,y,x) or (z,y,x,RGB)
- names (list of strings (or single)) – Names for each image
- Usage –
- ----- –
- the scroll interaction to work, the "scroller" object (For) –
- be returned to the calling scope (must) –
- fig, scroller = _scroll_plot4(images, names) (>>>) –
- fig.show() (>>>) –
elektronn2.utils.ptk_completions module¶
- Provides completions for the Python language, file sytem paths and a custom list of words
- for the ELEKTRONN2/Elektronn prompt_toolkit shell.
This module is mostly based on - https://github.com/jonathanslenders/ptpython/blob/master/ptpython/completer.py - https://github.com/jonathanslenders/ptpython/blob/master/ptpython/utils.py (at git revision 32827385cca65eabefccb06b56e4cf9d2c1e0120), which both are available under the following license (thanks, Jonathan and contributors!):
Copyright (c) 2015, Jonathan Slenders All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
- Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
- Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
- Neither the name of the {organization} nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
elektronn2.utils.utils_basic module¶
-
elektronn2.utils.utils_basic.
parallel_accum
(func, n_ret, var_args, const_args, proc=-1, debug=False)[source]¶
-
class
elektronn2.utils.utils_basic.
timeit
(*args, **kwargs)[source]¶ Bases:
elektronn2.utils.utils_basic.DecoratorBase
-
class
elektronn2.utils.utils_basic.
cache
(*args, **kwargs)[source]¶ Bases:
elektronn2.utils.utils_basic.DecoratorBase
-
class
elektronn2.utils.utils_basic.
AccumulationArray
(right_shape=(), dtype=<type 'numpy.float32'>, n_init=100, data=None, ema_factor=0.95)[source]¶ Bases:
object
-
data
¶
-
ema
¶
-
-
class
elektronn2.utils.utils_basic.
KDT
(n_neighbors=5, radius=1.0, algorithm='auto', leaf_size=30, metric='minkowski', p=2, metric_params=None, n_jobs=1, **kwargs)[source]¶ Bases:
sklearn.neighbors.unsupervised.NearestNeighbors
-
warning_shown
= False¶
-
-
class
elektronn2.utils.utils_basic.
DynamicKDT
(points=None, k=1, n_jobs=-1, rebuild_thresh=100, aniso_scale=None)[source]¶ Bases:
object
-
elektronn2.utils.utils_basic.
pickleload
(file_name)[source]¶ Loads all object that are saved in the pickle file. Multiple objects are returned as list.
-
elektronn2.utils.utils_basic.
picklesave
(data, file_name)[source]¶ Writes one or many objects to pickle file
- data:
- single objects to save or iterable of objects to save. For iterable, all objects are written in this order to the file.
- file_name: string
- path/name of destination file
-
elektronn2.utils.utils_basic.
h5save
(data, file_name, keys=None, compress=True)[source]¶ Writes one or many arrays to h5 file
- data:
- single array to save or iterable of arrays to save. For iterable all arrays are written to the file.
- file_name: string
- path/name of destination file
- keys: string / list thereof
- For single arrays this is a single string which is used as a name
for the data set.
For multiple arrays each dataset is named by the corresponding key.
If keys is
None
, the dataset names created by enumeration:data%i
- compress: Bool
- Whether to use lzf compression, defaults to
True
. Most useful for label arrays.
-
elektronn2.utils.utils_basic.
h5load
(file_name, keys=None)[source]¶ Loads data sets from h5 file
- file_name: string
- destination file
- keys: string / list thereof
- Load only data sets specified in keys and return as list in the order
of
keys
For a single key the data is returned directly - not as list If keys isNone
all datasets that are listed in the keys-attribute of the h5 file are loaded.
-
elektronn2.utils.utils_basic.
pretty_string_ops
(n)[source]¶ Return a humanized string representation of a large number.