2018-03-31 12:08

1 files changed, 135 insertions, 86 deletions

diff --git a/processing/chapter.tex b/processing/chapter.tex
index db8da3b..81886c2 100644
--- a/processing/chapter.tex
+++ b/processing/chapter.tex
@@ -23,19 +23,49 @@
 

 \clearpage

 

-From a data science perspective, CMDS has several unique challenges:

+CMDS takes a somewhat unique approach to instrumental science.  %

+There are not that many well-defined, well-trodden experimental paths.  %

+The basic ideas stay the same, but the real power is in the creativity and flexibility to tweak the

+experiment according to the particular question being asked.  %

+How, then could one go about making a data processing software package for CMDS?  %

+The package has to be flexible enough to accommodate the diversity of experiments, but still solid

+enough to be a foundational tool.  %

+

+When creating a toolkit for CMDS, there are several challenges worth considering:

 \begin{ditemize}

   \item Dimensionality of datasets can typically be greater than two, complicating

     \textbf{representation}.

-  \item Shape and dimensionality change...

-  \item Data can be large (over one million points).  % TODO: contextualize large (not BIG DATA)

+  \item Shape and dimensionality change, and relevant axes can be different from the scanned

+    dimensions.  %

+  \item Data can be awkwardly large-ish (several million pixels), and can become legitimately large

+    in numerical simulations.  %

+  \item There are no agreed-upon file formats for CMDS dataset storage.  %

 \end{ditemize}

-I have designed a software package that directly addresses these issues.  %

-

-WrightTools is a software package at the heart of all work in the Wright Group.  %

-

-% TODO: more intro

+The biggest challenge is to find a really good definition for what constitutes a CMDS dataset.  %

+Once understood, this common denominator can be enshrined into software and built upon.  %

+

+WrightTools is a software package written in Python, built using the excellent tools provided by

+the scientific Python collection of packages, especially Scipy and Numpy.  % TODO: cite cite cite

+WrightTools defines a universal file-format that is flexible enough to encompass the diversity of

+CMDS while still being entirely self-describing.  %

+This file format is based on the popular binary format ``HDF5''.  % TODO: cite

+This format allows for computers to interact with the arrays piece-by-piece in a very fast and

+reliable way, without loading the entire array in and out of memory.  %

+WrightTools piggybacks on this, allowing users to interact with legitimately large CMDS datasets

+without worrying about memory overflow.  %

+WrightTools takes a unique approach to representing CMDS data in array format, nick-named

+``semi-structure'', that allows for greater flexibility in representing CMDS in different

+coordinate spaces.  %

+

+WrightTools is written to be used in scripts and in the command line.  %

+It does not have any graphical components built in, except for the ability to generate plots using

+matplotlib.  % TODO: cite

+Being built in this way gives WrightTools users maximum flexibility, and allows for rapid

+collaborative development.  %

+It also allows other software packages to use WrightTools as a ``back-end'' foundational software,

+as has already been done in simulation and acquisition software created in the Wright Group.  %

 

+\clearpage

 \section{Introduction to WrightTools}  % ==========================================================

 

 WrightTools is written in Python, and endeavors to have a ``pythonic'', explicit and ``natural''

@@ -47,7 +77,7 @@ To use WrightTools, simply import:
 3.0.0

 \end{codefragment}

 I'll discuss more about how exactly WrightTools packaging, distribution, and instillation works in

-\autoref{sec:processing_distbribution}.

+\autoref{pro:sec:processing_distribution}.

 

 We can use the builtin Python function \python{dir} to interrogate the contents of the

 WrightTools package.  %

@@ -117,75 +147,86 @@ spectrum, and channels are each a particular kind of signal within that spectrum
 Typical variables might be \python{[w1, w2, w3, d1, d2]}, and typical channels

 \python{[pmt, pyro1, pyro2, pyro3]}.  %

 

-As an overview, the following lexicographically lists the attributes and methods of

-\python{Data}.  %

-\begin{ditemize}

-  \item method \python{collapse}: Collapse along one dimension in a well-defined way.

-  \item method \python{convert}: Convert all axes of a certain kind.

-  \item method \python{create_channel}: Create a new channel.

-  \item method \python{create_variable}: Create a new variable.

-  \item method \python{fullpath}

-  \item method \python{get_nadir}

-  \item method \python{get_zenith}

-  \item method \python{heal}

-  \item attribute \python{kind}

-  \item method \python{level}

-  \item method \python{map_variable}

-  \item attribute \python{natural_name}

-  \item attribute \python{ndim}

-  \item method \python{offset}

-  \item method \python{print_tree}

-  \item method \python{remove_channel}

-  \item method \python{remove_variable}

-  \item method \python{rename_channels}

-  \item method \python{rename_variables}

-  \item attribute \python{shape}

-  \item method \python{share_nans}

-  \item attribute \python{size}

-  \item method \python{smooth}

-  \item attribute \python{source}

-  \item method \python{split}

-  \item method \python{transform}

-  \item attribute \python{units}

-  \item attribute \python{variable_names}

-  \item attribute \python{variables}

-  \item method \python{zoom}

-\end{ditemize}

+\begin{table}

+  \begin{tabular}{c | c | l}

+    & type & description \\ \hline

+    \python{collapse} & method & Collapse along one dimension in a well-defined way. \\ \hline

+    \python{convert} & method & Convert all axes of a certain kind. \\ \hline

+    \python{create_channel} & method & Create a new channel. \\ \hline

+    \python{create_variable} & method & Create a new variable. \\ \hline

+    \python{fullpath} & attribute & \\ \hline

+    \python{get_nadir} & & \\ \hline

+    \python{get_zenith} & & \\ \hline

+    \python{heal} & & \\ \hline

+    \python{kind} & & \\ \hline

+    \python{level} & & \\ \hline

+    \python{map_variable} & & \\ \hline

+    \python{natural_name} & & \\ \hline

+    \python{ndim} & & \\ \hline

+    \python{offset} & & \\ \hline

+    \python{print_tree} & & \\ \hline

+    \python{remove_channel} & & \\ \hline

+    \python{remove_variable} & & \\ \hline

+    \python{rename_channels} & & \\ \hline

+    \python{shape} & & \\ \hline

+    \python{share_nans} & & \\ \hline

+    \python{size} & & \\ \hline

+    \python{smooth} & & \\ \hline

+    \python{source} & & \\ \hline

+    \python{split} & & \\ \hline

+    \python{transform} & & \\ \hline

+    \python{units} & & \\ \hline

+    \python{variable_names} & & \\ \hline

+    \python{variables} & & \\ \hline

+    \python{zoom} & & \\ \hline

+  \end{tabular}

+  \caption[Attributes and methods of Data]{

+    Key attributes and methods of data, lexicographically listed

+  }

+\end{table}

 

 Each data object contains instances of \python{Channel} and \python{Variable} which represent the

 principle multidimensional arrays.  %

 The following lexicographically lists the attributes of these instances.  %

 Certain methods and attributes are unique to only one type of dataset, and are marked as such.  %

-\begin{ditemize}

-  \item method \python{argmax}

-  \item method \python{argmin}

-  \item method \python{chunkwise}

-  \item method \python{clip}

-  \item method \python{convert}

-  \item attribute \python{full}

-  \item attribute \python{fullpath}

-  \item attribute \python{label} (variable only)

-  \item method \python{log}

-  \item method \python{log10}

-  \item method \python{log2}

-  \item method \python{mag}

-  \item attribute \python{major_extent} (channel only)

-  \item method \python{max}

-  \item method \python{min}

-  \item attribute \python{minor_extent} (channel only)

-  \item attribute \python{natural_name}

-  \item method \python{normalize} (channel only)

-  \item attribute \python{null} (channel only)

-  \item attribute \python{parent}

-  \item attribute \python{points}

-  \item attribute \python{signed} (channel only)

-  \item method \python{slices}

-  \item method \python{symmetric_root}

-  \item method \python{trim} (channel only)

-\end{ditemize}

+

 Channels and variables also support direct indexing / slicing using \python{__getitem__}, as

 discussed more in...  % TODO: where is it discussed more?

- 

+

+\begin{table}

+  \begin{tabular}{c | c | c | l}

+    & type & of & description \\ \hline

+    \python{argmax} & method & both & \\ \hline

+    \python{argmin} & & & \\ \hline

+    \python{chunkwise} & & & \\ \hline

+    \python{clip} & & & \\ \hline

+    \python{convert} & & & \\ \hline

+    \python{full} & & & \\ \hline

+    \python{fullpath} & & & \\ \hline

+    \python{label} & attribute & variable & \\ \hline

+    \python{log} & & & \\ \hline

+    \python{log10} & & & \\ \hline

+    \python{log2} & & & \\ \hline

+    \python{mag} & & & \\ \hline

+    \python{major_extent} & attribute & channel & \\ \hline

+    \python{max} & & & \\ \hline

+    \python{min} & & & \\ \hline

+    \python{minor_extent} & attribute & channel & \\ \hline

+    \python{natural_name} & & & \\ \hline

+    \python{normalize} & & channel & \\ \hline

+    \python{null} & & channel & \\ \hline

+    \python{parent} & & & \\ \hline

+    \python{points} & & & \\ \hline

+    \python{signed} & & channel & \\ \hline

+    \python{slices} & & & \\ \hline

+    \python{symmetric_root}

+    \python{trim} & & channel & \\ \hline

+  \end{tabular}

+  \caption[Attributes and methods of Channel and Variable.]{

+    Key attributes and methods of channel and variable, lexicographically listed

+  }

+\end{table}

+

 Axes are ways to organize data as functional of particular variables (and combinations thereof).  %

 The \python{Axis} class does not directly contain the respective arrays---it merely refers to the

 associated variables.  %

@@ -199,20 +240,28 @@ include \python{'w1'}, \python{'w1=wm'}, \python{'w1+w2'}, \python{'2*w1'}, \pyt
 Axes can be directly indexed / sliced into using \python{__getitem__}, and they support many of the

 ``numpy-like'' attributes.  %

 A lexicographical list of axis attributes and methods follows.

-\begin{ditemize}

-  \item attribute \python{full}

-  \item attribute \python{label}

-  \item attribute \python{natural_name}

-  \item attribute \python{ndim}

-  \item attribute \python{points}

-  \item attribute \python{shape}

-  \item attribute \python{size}

-  \item attribute \python{units_kind}

-  \item attribute \python{variables}

-  \item method \python{convert}

-  \item method \python{min}

-  \item method \python{max}

-\end{ditemize}  % TODO: actually lexicographical

+

+

+\begin{table}

+  \begin{tabular}{c | c | l}

+    & type & description \\ \hline

+    \python{full} & & \\ \hline

+    \python{label} & & \\ \hline

+    \python{natural_name} & & \\ \hline

+    \python{ndim} & & \\ \hline

+    \python{points} & & \\ \hline

+    \python{shape} & & \\ \hline

+    \python{size} & & \\ \hline

+    \python{units_kind} & & \\ \hline

+    \python{variables} & & \\ \hline

+    \python{convert} & & \\ \hline

+    \python{min} & & \\ \hline

+    \python{max} & & \\ \hline

+  \end{tabular}

+  \caption[Attributes and methods of Axis.]{

+    Key attributes and methods of axis, lexicographically listed

+  }

+\end{table}

 

 \section{Creating a data object}  % ===============================================================

 

@@ -309,7 +358,7 @@ Conceptually, it behaves like a folder in a traditional file-system.  %
 

 The primary attributes and methods of \python{Collection} are

 \begin{ditemize}

-  \item attribute item_names

+  \item attribute \python{item_names}

   \item attribute \python{fullpath}

 \end{ditemize}

 % TODO: finish adding attributes and methodsd