Cells

The Cell is the core concept that one must grok to use ecto. Think of a cell as a small self contained well formed unit of processing machinery. Each cell’s job is to take some number of inputs, and transform them into some number of outputs. Of course, parameters may have some effect on this transformation, and cells may each have their own state that they alone govern.

Let us examine a c++ construct, a common functor:

struct Printer
{
  Printer(const std::string& prefix, const std::string& suffix)
      :
        prefix_(prefix),
        suffix_(suffix)
  {
  }
  void
  operator()(std::ostream& out, const std::string& message)
  {
    out << prefix_ << message << suffix_;
  }
  std::string prefix_, suffix_;
};

A c++ function object.

Functors, function objects, are a reasonable way of encapsulating self contained functionality to operate on a set of data, and mesh well with the stl algorithms. The operator() may be abused by templated code to give way to genericity.

The ecto Cell is similar, in that it gives ecto a common unit with which to work. Here is the ecto equivalent of the above functor, slightly more verbose of course:

#include <ecto/ecto.hpp>
#include <ecto/registry.hpp>
#include <iostream>
#include <string>
using ecto::tendrils;
namespace overview
{
  struct Printer01
  {
    static void
    declare_params(tendrils& params)
    {
      params.declare<std::string>("prefix", "A string to prefix printing with.", "start>> ");
      params.declare<std::string>("suffix", "A string to append printing with.", " <<stop\n");
    }
    static void
    declare_io(const tendrils& params, tendrils& inputs, tendrils& outputs)
    {
      inputs.declare<std::string>("message", "The message to print.");
    }
    void
    configure(const tendrils& params, const tendrils& inputs, const tendrils& outputs)
    {
      params["prefix"] >> prefix_;
      params["suffix"] >> suffix_;
    }
    int
    process(const tendrils& inputs, const tendrils& outputs)
    {
      std::cout << prefix_ << inputs.get<std::string>("message") << suffix_;
      return ecto::OK;
    }
    std::string prefix_, suffix_;
  };
}
ECTO_CELL(ecto_overview, overview::Printer01, "Printer01",
          "A simple stdout printer with prefix and suffix parameters.");

A typical ecto::cell.

The verbosity is a feature of ecto, in that each cell exposes as much information as it can to the ectosphere. At runtime, ecto may ask the cell for documentation, types, and semantics. These features are what enable the type-safety in a graph of cells, or auto-completion from an ipython context.

Examination of the Cell

A Cell defines some set of parameters, inputs and outputs statically. And each cell created from this static definition holds its own state, and may operate on it’s parameters,inputs and outputs at specific moments throughout it’s lifetime. Also the outside world may examine and manipulate the cells tendrils.

Let us look at a graphical representation of the Cell written above:

A graphical representation of a ecto::cell. In green are the inputs, blue parameters, yellow cell type. This cell has no outputs.

The parameters, inputs, and outputs of a cell all share the same type, tendril. Tendrils are mappings between strings, and lazily typed objects. These are how the cell communicates with the rest of the system. The reason for choosing a runtime typed object like the tendril instead of a compile time typed object like a boost::tuple is that it allows for ecto to be type ignorant, as the data held by the tendril has really no effect on how a graph executes or the python interfaces. Ecto is a plugin based architecture, and so can not be header only, strictly compile time typed.

Optional interface

The most basic of cells would be:

#include <ecto/ecto.hpp>
namespace overview
{
  struct NopCell
  {
  };
}
ECTO_CELL(ecto_overview, overview::NopCell, "NopCell",
          "A cell that can't do anything.");

NopCell

Brief doc

A cell that can’t do anything.

Each cell may or may not implement the following functions:

#include <ecto/ecto.hpp>
using ecto::tendrils;
namespace overview
{
  struct InterfaceCell
  {
    static void
    declare_params(tendrils& params);

    static void
    declare_io(const tendrils& params, tendrils& in, tendrils& out);

    void
    configure(const tendrils& params, const tendrils& in, const tendrils& out);

    int
    process(const tendrils& in, const tendrils& out);
  };
}
ECTO_CELL(ecto_overview, overview::InterfaceCell, "InterfaceCell",
          "A cell cell implementing the entire ecto interface");

The cell interface functions.

A note on where to implement

Notice that in most examples we implement the functions for cells inline in the declaration of the struct. However there is nothing stopping you from writing the implementation elsewhere.

However if you do implement any of the methods in the cell interface, be sure that their signatures match the above specification.

A peek under the covers

But wait, how does ecto know anything about my cell? You must be full of black magic! Where is the inheritence and polymorphism?

—Egon

You might be thinking these kinds of things... Well the registration with ecto occurs in the macro ECTO_CELL. This macro does some amount of extra fanciness, but in the end it takes your struct and does something similar to the following simplified example:

struct cell
{
  virtual ~cell(){}
  virtual void declare_params() = 0;
  virtual int process() = 0;
  //... other interface functions
  ecto::tendrils params_, inputs_, outputs_;
};

template<typename YourCell>
struct cell_ : cell
{
  void declare_params()
  {
    YourCell::declare_params(params_);
  }

  void configure()
  {
    if(!thiz_)
    {
      thiz_ = new YourCell();
      thiz_->configure();
    }
  }
  //... dispatch other functions...
  YourCell* thiz_;
}

cell_<InterfaceCell> c;

The real implementation uses SFINAE to enable optional implementation of the interface functions. The macro ECTO_CELL also constructs python bindings for your cell, and generates RST formated doc strings from the static declaration parameter and io functions. The above sample should be referred to as the essence of the technique, rather than the exact implementation. This technique gives a certain amount of opaqueness to the client cell implementers, and provides ecto with a flexible entry point for implementation details.

Doing work

Let us take the Printer from above and use it from python. The following python script demonstrates the python interface of our cell that ecto provides for free(assuming you used the macro and followed the cell interface).

#!/usr/bin/python 
from ecto.ecto_overview import Printer01

print Printer01.__doc__

printer = Printer01(prefix='... ', suffix=' ...\n')

printer.inputs.message = 'TFJ'

printer.process()

The script, when run will give the following output:

No module named PySide.QtCore
A simple stdout printer with prefix and suffix parameters.

Parameters:
 - prefix [std::string] default = start>> 
    A string to prefix printing with.

 - suffix [std::string] default =  <<stop

    A string to append printing with.

Inputs:
 - message [std::string]
    The message to print.


... TFJ ...

Doc Generation

Notice the line print Printer01.__doc__, every ecto cell gets this for free based on the docstrings that were written in the static declare_* functions. This is a class level attribute, and is one of the justifications for having the declare_* functions be static. At import time, the __doc__ strings are generated, and it is important that you write cells that will not crash during import.

Another cool aspect of documentation generation is its full integration into sphinx docs. Placing the following command in sphinx:

.. ectocell:: ecto.ecto_overview Printer01

Produces:

Printer01

Brief doc

A simple stdout printer with prefix and suffix parameters.

Parameters

• prefix   type: std::string    not required   default: start>>

  A string to prefix printing with.

• suffix   type: std::string    not required   default: <<stop

  A string to append printing with.

Inputs

• message   type: std::string   

  The message to print.

Also from an interactive python prompt, the following should produce useful output:

>>> from ecto.ecto_overview import Printer01
>>> help(Printer01)

For more detailed information, refer to Automatic Documentation and Ecto’s Sphinx Extensions.

Cell construction

Let us take a closer look at the python sample code. Take the cell construction line:

printer = Printer01(prefix='... ', suffix=' ...\n')

A typical cell constructor call from python. Notice mapping from keyword arguments to the parameters declared in Printer.

Every Cell gets a constructor that has the following python signature:

Cell([Cell.type_name()], [param1=...],[param2=...])

When is configure called?

Your cell is not actually allocated(constructed) until a call to process or right before a scheduler executes your processing graph, see schedulers.

This is important, as python may be used to quickly iterate on your processing graph structure, and heavy allocation (and possible crashes) will be lazily evaluated when the Cell is actually used to do work.

This may be abused for documentation, prototyping, and GUI applications.

The first optional non-keyword argument is the cell’s instance name. The instance name is useful for debug and graph display purposes. Each parameter that was declared by the cell may be initialized as a keyword argument. Other advanced keyword arguments may exist, that are defined by ecto, such as Strands: preventing cells from running concurrently.

The constructor of the cell calls declare_params, sets the cell’s parameters with any supplied by keyword arguments, then precedes to call declare_io, with the parameter values set by the keyword arguments. It is important to note that when the constructor returns a new cell, the constructor of the struct, e.g. Printer01, has not yet been called, also, configure has not yet been called.

Configuration and Processing

If it has a boost::python binding, life is wonderful

You may notice the assignment of a string to tendril from python, even though the tendril was declared with a c++ type. Its all good as long as you have boost::python bindings. See Tendril Conversions for an explanation.

If you don’t have bindings, not to worry, you just won’t be able to manipulate your tendrils from python... Oh and you get python bindings for free for all the basic types.

So even though we only have a shell of a cell, we can still manipulate its tendrils. Lets set the input tendril, called message, to a custom string. Then we’ll call process.

printer.inputs.message = 'TFJ'
printer.process()

Now, before the call to process our Printer01 hasn’t actually been allocated, but all of its tendrils, inputs, outputs, and parameters are accessible from python. During the call to process, the Printer01 is allocated as necessary, the configure function is called, any parameter-callbacks are triggered if parameters have changed value, and then process is executed.

Cell Python Interface

Every cell exposes a bit of functionality to python, which may be useful to know.

class Cell

A sketch of the cell python interface.

(Tendrils)parameters

The parameters of the cell, initialized by the cell’s constructor.

(Tendrils)inputs

The inputs may be initialized with dependence on the parameters but are valid and free game in python.

(Tendrils)outputs

Same as inputs.

configure()

Allocates the underlying cell, say Printer01, and dispatches configure. Will not have any effect the second time configure is called.

process()

May call configure if it has not already been called. Dispatches a call to process in the underlying cell, e.g. Printer::process(...)

Cell Life Cycle

The life cycle of cell is important to keep in mind.

• declare_params and declare_io are static functions and will be called in order to initialize the cell’s parameters,inputs, and outputs.
• configure will be called once right after the construction of the cell.
• process will be called any number of times, with parameter callbacks being triggered prior to its execution.
• When all references to the cell are gone, i.e. it has gone out of scope, then it may be deleted.