1 files changed, 80 insertions, 12 deletions
diff --git a/processing/chapter.tex b/processing/chapter.tex
index 72d1d27..fa449fe 100644
--- a/processing/chapter.tex
+++ b/processing/chapter.tex
@@ -168,9 +168,10 @@ Channels and variables also support direct indexing / slicing using \python{__ge
 discussed more in...  % TODO: where is it discussed more?
  
 Axes are ways to organize data as functional of particular variables (and combinations thereof).  %
-The \python{Axis} class does not directly contain the respective arrays---it refers to the
+The \python{Axis} class does not directly contain the respective arrays---it merely refers to the
 associated variables.  %
 The flexibility of this association is one of the main new features in WrightTools 3.  %
+It enables data transformation, discussed in section ...  % TODO: link to section
 Axis expressions are simple human-friendly strings made up of numbers and variable
 \python{natural_name}s.  %
 Given 5 variables with names \python{['w1', 'w2', 'wm', 'd1', 'd2']}, example valid expressions
@@ -229,21 +230,64 @@ a parameter-free from-function.  %
 There are two problems:
 \begin{ditemize}
   \item Dimensionality limitation to individual files (1D for KENT, 2D for COLORS).
-  \item Lack of self-describing metadata.
+  \item Lack of self-describing metadata (headers).
 \end{ditemize}
 The way that WrightTools handles data creation for these file-types deserves special discussion.  %
 
-Firstly, WrightTools contains hardcoded column information for each filetype...
-For COLORS...  % TODO
-
-Secondly, WrightTools accepts a list of files which it stacks together to form a single large
-array.  %
-
-Finally, the \python{wt.kit.discover_dimensions} function is called.  %
-This function does its best to recognize the parameters of the original scan...  % TODO
+Firstly, WrightTools contains hardcoded column information for each filetype.
+Data from Kent Meyer's ``picosecond control'' software had consistent columns over the lifetime of
+the software, so only one dictionary is needed to store these correspondences.  %
+Skye Kain's ``COLORS'' software used at least 7 different formats, and unfortunately these format
+types were not fully documented.  %
+WrightTools attempts to guess the COLORS data format by counting the number of columns.  %
+
+Because these file-types are dimensionality limited, there are many acquisitions that span over
+multiple files.  %
+COLORS offered an explicit queue manager which allowed users to repeat the same 2D scan (often a
+Wigner scan) many times at different coordinates in non-scanned dimensions.  %
+ps\_control scans were done more manually.  %
+To account for this problem of multiple files spanning a single acquisition, the functions
+\python{from_COLORS} and \python{from_KENT} optionally accept \emph{lists} of filepaths.  %
+Inside the function, WrightTools simply appends the arrays from all given files into one long array
+with many more rows.  %
+
+The final and most challenging problem of parameter-free importing for these filetypes is
+\emph{dimensionality recognition}.  %
+Because the files contain no metadata, the shape and coordinates of the original acquisition must
+be guessed by simply inspecting the columnar arrays.  %
+In general, this problem can become very hard.  %
+Luckily, each of these previous instrumental software packages was only used on one instrument with
+limited flexibility in acquisition type, so it is possible to make educated guesses for almost all
+acquisitions.  %
+
+The function \python{wt.kit.discover_dimensions} handles the work of dimensionality recognition for
+both COLORS and ps\_control arrays.  %
+This function may be used for more filetypes in the future.  %
+Roughly, the function does the following:
+\begin{denumerate}
+  \item Remove dimensions containing nan(s).
+  \item Find which dimensions are equal (within tolerance), condense into single dimensions.
+  \item Find which dimensions are scanned (move beyond tolerance).
+  \item For each scanned dimension, find how many unique (outside of toelerance) points were taken.
+  \item Linearize each scanned dimension between smallest and largest unique point.
+  \item Return scanned dimension names, column indices and points.
+\end{denumerate}
+The \python{from_COLORS} and \python{from_KENT} functions then linearly interpolate each row in the
+channels onto the grid defined by \python{discover_dimensions}.  %
+This interpolation uses \python{scipy.interpolate.griddata}, which in turn relies upon the C++
+library Qhull.  %
+
+This strategy can be copied in the future if other non-self-describing data sources are added into
+WrightTools.  %
 
 \subsubsection{From directory}
 
+The \python{wt.collection.from_directory} function can be used to automatically import all of the
+data sources in an entire directory tree.  %
+It returns a WrightTools collection with the same internal structure of the directory tree, but
+with WrightTools data objects in the place of raw data source files.  %
+Users can configure which files are routed to which from-function.  %
+
 % TODO (also document on wright.tools)
 
 \subsection{Math}  % ------------------------------------------------------------------------------
@@ -252,9 +296,33 @@ Now that we know the basics of how the WrightTools \python{Data} class stores da
 some data manipulation.  %
 Let's start with some elementary algebra.  %
 
-\subsubsection{In place operators}
+\subsubsection{In-place operators}
+
+In Python, operators are symbols that carry out some computation.  %
+Consider the following:
+\begin{codefragment}{python, label=abcdefg}
+>>> import numpy as np
+>>> a = np.array([4, 5, 6])
+>>> b = np.array([-1, -2, -3])
+>>> c = a + b
+>>> c
+array([3, 3, 3])
+\end{codefragment}
+Here, \python{a} and \python{b} are operands and \python{+} is an operator.  %
+When used in this simple way, operators typically create and return a \emph{new} object in the
+computers memory.  %
+We can verify this by using Python's built-in \python{id} function on the objects created in
+\ref{abcdefg}.  %
+\begin{codefragment}{python}
+>>> id(a), id(b), id(c)
+(139712529580400, 139712333712320, 139712333713040)
+\end{codefragment}
+This is usually fine, but sometimes the operands are unwieldy large objects that take a lot of
+memory to store.  %
+In other cases operators are used millions of times such that, used as above, millions of new
+arrays will be created.  %
+One way to avoid these problems is to use \emph{in-place} operators
 
-Operators are...  % TODO
 Because the \python{Data} object is mostly stored outside of memory, it is better to do
 in-place... % TODO