tag:support.nodebox.net,2012-11-01:/discussions/general-discussion/14667-the-origins-of-nodebox-3NodeBox: Discussion 2017-12-05T06:13:04Ztag:support.nodebox.net,2012-11-01:Comment/379782722015-09-21T01:03:10Z2015-09-21T01:03:10ZThe Origins of Nodebox 3<div><p>While randomly surfing the web I happened across the following
May 2011 blog on Fredrick's enigmeta site:</p>
<p><a href="http://www.enigmeta.com/2011/05/01/nodebox-2-is-a-graph/">http://www.enigmeta.com/2011/05/01/nodebox-2-is-a-graph/</a></p>
<p>It's about how NodeBox 2's functional approach differs from
imperative languages like Processing and the problems you run into
when you try to convert Processing to a visual programming
language. It ends with Frederick saying "Figuring out these
semantics will be part of my research project."</p>
<p>I found this fascinating. It gels with the ruminations in my own
recent blog post about why visual programming works better for
functional languages. I assume that this line of thought is what
led to NodeBox 3. These concepts helped me understand why NodeBox 3
succeeds where so many other approaches have failed.</p>
<p>Four years have passed since that post was written. Looking back
now, Frederick, do you have any comments about lessons learned and
what open questions remain?</p>
<p>John</p></div>johntag:support.nodebox.net,2012-11-01:Comment/379782722015-09-21T08:48:32Z2015-09-21T08:48:32ZThe Origins of Nodebox 3<div><p>Yes, in fact that will be part of my PhD :-)</p>
<h2><a href="#state-processing" class="anchor" name="state-processing"></a>State /
Processing</h2>
<p>I don't think we'll ever be able to marry the stateful model of
Processing with the functional model of NodeBox.</p>
<p>The idea when writing that blog post was about allowing users to
write nodes in Processing, and have them integrate with the rest of
the system. However, as is clear, since the functions in Processing
set state or draw stuff directly, there is nothing to integrate. We
could still choose to use Processing as a drawing API, but this
negates all the benefits of using Processing (you can't really wrap
existing Processing code in nodes, for example).</p>
<h2><a href="#variation" class="anchor" name="variation"></a>Variation</h2>
<p>If your whole model is functional, how do you generate
variation? How do you do this typical <code>for
(i=0;i<size;i++)</code> loop where i will determine the size of
the shape, for example?This is where NodeBox 2 and NodeBox 3 differ
fundamentally. In NodeBox 2, we used this principle called "copy
stamping" (inspired by Houdini) where we would travel back up to
the nodes to modify them, then go down again to render them. It
requires users to input expressions to generate variation. The idea
was for the expression to be a gateway drug into "real" coding.
However, the expressions just became unwieldy.</p>
<p>In NodeBox 3, as you know, we moved to list matching as a way of
generating variations. If you want alternating colors for a grid of
rectangles, you provide a list of colors that you want to alternate
between and a list of points for the grid, and NodeBox will match
them up. This works much better in practice, especially for data
visualisation.</p>
<h2><a href="#feedback-loops" class="anchor" name="feedback-loops"></a>Feedback loops</h2>
<p>Let's think about the stateful nature of feedback loops like
boids. Since NodeBox is functional, the side effects of a function
(like the new position of each boid) can't be stored inside of the
node. Each node should be a deterministic operation, where the same
input always leads to the same output.</p>
<p>This is an issue with functional programming in general, and we
can look towards other functional languages to see how they deal
with it. Which leads us to.... monads!</p>
<p>The Haskell <a href="https://en.wikibooks.org/wiki/Haskell/Understanding_monads/State">state
monad</a> is a solution to our issue. In NodeBox, we can implement
this quite cleanly using a "state" port that takes in the
<em>previous</em> value. This works well for small examples like
counters, etc. Bigger networks, with multiple independent states,
are much trickier and often impossible.</p>
<p>Another issue is with subnetworks. If the state is in the
subnetwork, we don't want to use the same state every time. We
really want a different state "instance" for each run through the
subnetwork. We can solve this (though we haven't yet) using a
path-based approach, where we tag on the path that we've travelled
through the network. Something like
<code>root.subnet1[5].counter</code>. (Again, this is fictive since
we haven't implemented this yet).</p>
<p>Note that monads aren't the only solution. Clojure, for example,
manages things fine without thinking about monads. However, in
those cases the code you write looks more imperative than
functional. Maybe this is a solution as well: allow for a bit of
imperative code to sneak in to fill in the conceptual gaps of
functional programming.</p>
<h2><a href="#subnetworks" class="anchor" name="subnetworks"></a>Subnetworks</h2>
<p>Which brings me to subnetworks. The way nodes are processed in
NodeBox Live is nearly identical to how we do it in NodeBox 3,
<em>except</em> for subnetworks.</p>
<p>What I've found is that each time we invent our own visual thing
that has no analog in textual coding (like subnetworks, or copy
stamping), we get into trouble. A better solution is to use systems
that we have in the textual world and appropriate them. That's why
we use regular functions in NodeBox Live instead of
subnetworks.</p>
<p>This gives us some nice properties: we have a list of functions
that we can re-use all over the place. Functions can be written
visually and functionally, and they expose the same interface
(multiple arguments in, one argument out). In fact, visual
functions get compiled to JavaScript functions (we still need the
NodeBox runtime, but we can mostly get rid of that as well if we
want to).</p>
<h2><a href="#external-input" class="anchor" name="external-input"></a>External input</h2>
<p>This all works fine when we're dealing with inputs generated
inside the computer. External inputs still present a problem.</p>
<p>NodeBox "pulls" data from the bottom node to the top node, using
as much cached data as possible. Since external input comes from
the top, we have an issue. We would need to make nodes that take in
external input dirty, then propagate that dirtiness to all
downstream nodes, until we reach the rendered node. We'll also have
to figure out a way to make these external inputs available: a
simple solution is just to expose a node with no inputs. We already
do this for mouse and OSC.</p>
<h2><a href="#external-output" class="anchor" name="external-output"></a>External
output</h2>
<p>We would also like to be able to let our networks have effect on
the outside world: think about steering an Arduino robot, or
sending OSC events to PD,... This requires our networks to have
side effects, which goes against the fundamental nature of
functional programming. Specifically, it breaks the single
"rendered node" approach: what node should the rendered node be? If
the "send OSC" node is rendered, we don't see any visual
output.</p>
<p>We currently don't have good solutions for this. One hack is to
put a "send OSC" node in a combine node. It doesn't return anything
but it still gets executed. However, this is a big hack and one I'm
not a fan of. The "order number" from the NodeBox 2 post might be a
solution, albeit a quite difficult one.</p>
<h2><a href="#conclusion" class="anchor" name="conclusion"></a>Conclusion</h2>
<p>There are probably more issues we have to tackle, but they all
come back to trying to pigeonhole functional programming onto a
computer architecture that is fundamentally imperative. Note that
I'm not bashing on FP: functional programming is very attractive
for most of the things we'd like to do with NodeBox, like
visualisations or generative designs. However, some issues lend
themselves much better to imperative code than to functional code.
The processor in our machine is fundamentally about manipulating
memory, and hence, changing state. Any deviation from that
imperative model will require thinking about how to manage state
within that model.</p></div>Frederik De Blesertag:support.nodebox.net,2012-11-01:Comment/379782722015-09-24T10:18:07Z2015-09-24T10:18:07ZThe Origins of Nodebox 3<div><p>Thanks, Fredrick! This was very interesting! It seems to me like
you've already earned your PhD several times over.</p>
<p>A little question and a big question...</p>
<p>Little: If you do implement state monads, would that finally
allow us to do L-systems and true recursion in NodeBox? How would
that work?</p>
<p>Big: Will the improvements you make to NodeBox Live also appear
in NodeBox 3 some day so that the web and native versions continue
to evolve together and compliment one another? Or will NodeBox 3 go
into maintenance mode and gradually fade from the scene like
NodeBox 2?</p>
<p>I like some of the improvements you've made to NodeBox Live, and
you've done an heroic job with the UI, but I don't find myself
drawn to it. It's not just the rough edges and gaps present in any
beta product. For me the overall user experience doesn't feel as
solid, clean, uncluttered, and responsive as the native app.</p>
<p>I have the same reaction to pretty much all web apps. I dislike
web versions of word processors, spreadsheets, mail apps, etc. Even
the best of them still seem clunky and the extra chrome and cruft
of a web browser is distracting. Especially for a drawing tool
which lives or dies by the ease with with you can scrub and pan and
zoom and drag and group select and arrange your workspace across
multiple monitors it's hard to imagine that I'd ever prefer web to
native.</p>
<p>What I really want is to continue coding on my Mac/PC (a native
iPad Pro app would be even better!) and then upload to NodeBox Live
when I'm ready to share. The web is great for sharing information,
but native still rules for pure creation.</p>
<p>Will I have to choose or can I have both?</p>
<p>John</p></div>johntag:support.nodebox.net,2012-11-01:Comment/379782722015-11-27T12:29:53Z2015-11-27T12:29:53ZThe Origins of Nodebox 3<div><p>Hi John,</p>
<p>This answer is long overdue -- sorry for that!</p>
<p>What you mentioned (work on desktop/ipad, export to web) is
certainly an option we're looking into.</p>
<p>Here's my main idea: if we see platforms as a continuum, you
could see "the web" on one side, with a huge reach and a
programming language (JavaScript) that's available everywhere. It
is fast enough for most tasks, but falls short for real-time,
interactive graphics and installations (think: Oculus VR,
multi-megabyte realtime input streams), and on mobile this speed
difference is much more pronounced.</p>
<p>On the other side is C/C++. It is super-fast. It is native to
iOS, Mac, Windows, Linux (and Android, using the NDK); and thanks
to Emscripten it also runs on the web.</p>
<p>Then there's the stuff in between. Java (which is what NodeBox 3
is written in) is an okay language, it is quite fast, but it's hard
to interface with native stuff, or to get running on iOS. Python
works very well on servers and in your own projects, but doesn't
have a good desktop or mobile story (same for other dynamic
programming languages such as Ruby).</p>
<p>Remember, we have a very small team so we don't want to spread
ourselves too thin. But I do believe we need these two sides
covered: one side that runs on the web (NodeBox Live), and one side
that runs where the web is not ideal (phones, Raspberry Pi,
physical installations).</p>
<p>So, to answer your question, NodeBox 3 will not disappear, but
it will transform into something, an "app" that runs on desktops
and smaller devices. I'm not sure if the native web version and the
C version will ever be compatible, but we'll provide export options
from C (through Emscripten for example).</p>
<p>We've already started development of this C version, and have
something running on iPad and desktop. But it's still a long way
off. PhD first...</p>
<p>Hopefully that answers your question. Let me know if you have
any additional remarks.</p>
<p>F</p></div>Frederik De Blesertag:support.nodebox.net,2012-11-01:Comment/379782722015-11-28T21:22:18Z2015-11-28T21:22:18ZThe Origins of Nodebox 3<div><p>Thank you, Frederick. Very helpful.</p>
<p>I am <em>delighted</em> to hear you are exploring a native
version for the iPad. I now have an iPad Pro - which might just be
the perfect device for NodeBox. Let me know if you need a beta
tester!</p>
<p>As for your other ambitious plans I wonder if you are already
stretched too thin. Simply maintaining (and improving!) NodeBox 3
across three different platforms is a full time job, let alone
NodeBox Live and NodeBox C++. And as I'm sure you know only too
well, a PhD can eat years all by itself and drives some people into
the loony bin.</p>
<p>I attended the EyeO conference this spring and got a chance to
hear and interact with Ben Fry and Casey Reas. They are both proud
of Processing but complained that it became an all-consuming chore
to maintain. Their solution was to create a foundation to oversee
and nurture multiple extensions of Processing (p5.js,
Processing.py).</p>
<p>I suspect you are not in a position to snap your fingers and
make a foundation appear. But maybe it is time to search for new
patrons or write a grant or do something to create a more formal
and sustained support system. I worry that without more help
NodeBox may evolve too slowly to attract the full audience it
deserves.</p>
<p>Keep up the good work and good luck on your PhD!</p>
<p>John</p></div>johntag:support.nodebox.net,2012-11-01:Comment/379782722015-12-10T19:19:22Z2015-12-10T19:19:22ZThe Origins of Nodebox 3<div><p>The foundation is a good idea! We'll discuss with the team.</p></div>Frederik De Bleser