The Origins of Nodebox 3
While randomly surfing the web I happened across the following May 2011 blog on Fredrick's enigmeta site:
http://www.enigmeta.com/2011/05/01/nodebox-2-is-a-graph/
It's about how NodeBox 2's functional approach differs from imperative languages like Processing and the problems you run into when you try to convert Processing to a visual programming language. It ends with Frederick saying "Figuring out these semantics will be part of my research project."
I found this fascinating. It gels with the ruminations in my own recent blog post about why visual programming works better for functional languages. I assume that this line of thought is what led to NodeBox 3. These concepts helped me understand why NodeBox 3 succeeds where so many other approaches have failed.
Four years have passed since that post was written. Looking back now, Frederick, do you have any comments about lessons learned and what open questions remain?
John
Keyboard shortcuts
Generic
| ? | Show this help |
|---|---|
| ESC | Blurs the current field |
Comment Form
| r | Focus the comment reply box |
|---|---|
| ^ + ↩ | Submit the comment |
You can use Command ⌘ instead of Control ^ on Mac
Support Staff 1 Posted by Frederik De Ble... on 21 Sep, 2015 08:48 AM
Yes, in fact that will be part of my PhD :-)
State / Processing
I don't think we'll ever be able to marry the stateful model of Processing with the functional model of NodeBox.
The idea when writing that blog post was about allowing users to write nodes in Processing, and have them integrate with the rest of the system. However, as is clear, since the functions in Processing set state or draw stuff directly, there is nothing to integrate. We could still choose to use Processing as a drawing API, but this negates all the benefits of using Processing (you can't really wrap existing Processing code in nodes, for example).
Variation
If your whole model is functional, how do you generate variation? How do you do this typical
for (i=0;i<size;i++)loop where i will determine the size of the shape, for example?This is where NodeBox 2 and NodeBox 3 differ fundamentally. In NodeBox 2, we used this principle called "copy stamping" (inspired by Houdini) where we would travel back up to the nodes to modify them, then go down again to render them. It requires users to input expressions to generate variation. The idea was for the expression to be a gateway drug into "real" coding. However, the expressions just became unwieldy.In NodeBox 3, as you know, we moved to list matching as a way of generating variations. If you want alternating colors for a grid of rectangles, you provide a list of colors that you want to alternate between and a list of points for the grid, and NodeBox will match them up. This works much better in practice, especially for data visualisation.
Feedback loops
Let's think about the stateful nature of feedback loops like boids. Since NodeBox is functional, the side effects of a function (like the new position of each boid) can't be stored inside of the node. Each node should be a deterministic operation, where the same input always leads to the same output.
This is an issue with functional programming in general, and we can look towards other functional languages to see how they deal with it. Which leads us to.... monads!
The Haskell state monad is a solution to our issue. In NodeBox, we can implement this quite cleanly using a "state" port that takes in the previous value. This works well for small examples like counters, etc. Bigger networks, with multiple independent states, are much trickier and often impossible.
Another issue is with subnetworks. If the state is in the subnetwork, we don't want to use the same state every time. We really want a different state "instance" for each run through the subnetwork. We can solve this (though we haven't yet) using a path-based approach, where we tag on the path that we've travelled through the network. Something like
root.subnet1[5].counter. (Again, this is fictive since we haven't implemented this yet).Note that monads aren't the only solution. Clojure, for example, manages things fine without thinking about monads. However, in those cases the code you write looks more imperative than functional. Maybe this is a solution as well: allow for a bit of imperative code to sneak in to fill in the conceptual gaps of functional programming.
Subnetworks
Which brings me to subnetworks. The way nodes are processed in NodeBox Live is nearly identical to how we do it in NodeBox 3, except for subnetworks.
What I've found is that each time we invent our own visual thing that has no analog in textual coding (like subnetworks, or copy stamping), we get into trouble. A better solution is to use systems that we have in the textual world and appropriate them. That's why we use regular functions in NodeBox Live instead of subnetworks.
This gives us some nice properties: we have a list of functions that we can re-use all over the place. Functions can be written visually and functionally, and they expose the same interface (multiple arguments in, one argument out). In fact, visual functions get compiled to JavaScript functions (we still need the NodeBox runtime, but we can mostly get rid of that as well if we want to).
External input
This all works fine when we're dealing with inputs generated inside the computer. External inputs still present a problem.
NodeBox "pulls" data from the bottom node to the top node, using as much cached data as possible. Since external input comes from the top, we have an issue. We would need to make nodes that take in external input dirty, then propagate that dirtiness to all downstream nodes, until we reach the rendered node. We'll also have to figure out a way to make these external inputs available: a simple solution is just to expose a node with no inputs. We already do this for mouse and OSC.
External output
We would also like to be able to let our networks have effect on the outside world: think about steering an Arduino robot, or sending OSC events to PD,... This requires our networks to have side effects, which goes against the fundamental nature of functional programming. Specifically, it breaks the single "rendered node" approach: what node should the rendered node be? If the "send OSC" node is rendered, we don't see any visual output.
We currently don't have good solutions for this. One hack is to put a "send OSC" node in a combine node. It doesn't return anything but it still gets executed. However, this is a big hack and one I'm not a fan of. The "order number" from the NodeBox 2 post might be a solution, albeit a quite difficult one.
Conclusion
There are probably more issues we have to tackle, but they all come back to trying to pigeonhole functional programming onto a computer architecture that is fundamentally imperative. Note that I'm not bashing on FP: functional programming is very attractive for most of the things we'd like to do with NodeBox, like visualisations or generative designs. However, some issues lend themselves much better to imperative code than to functional code. The processor in our machine is fundamentally about manipulating memory, and hence, changing state. Any deviation from that imperative model will require thinking about how to manage state within that model.
Support Staff 2 Posted by john on 24 Sep, 2015 10:18 AM
Thanks, Fredrick! This was very interesting! It seems to me like you've already earned your PhD several times over.
A little question and a big question...
Little: If you do implement state monads, would that finally allow us to do L-systems and true recursion in NodeBox? How would that work?
Big: Will the improvements you make to NodeBox Live also appear in NodeBox 3 some day so that the web and native versions continue to evolve together and compliment one another? Or will NodeBox 3 go into maintenance mode and gradually fade from the scene like NodeBox 2?
I like some of the improvements you've made to NodeBox Live, and you've done an heroic job with the UI, but I don't find myself drawn to it. It's not just the rough edges and gaps present in any beta product. For me the overall user experience doesn't feel as solid, clean, uncluttered, and responsive as the native app.
I have the same reaction to pretty much all web apps. I dislike web versions of word processors, spreadsheets, mail apps, etc. Even the best of them still seem clunky and the extra chrome and cruft of a web browser is distracting. Especially for a drawing tool which lives or dies by the ease with with you can scrub and pan and zoom and drag and group select and arrange your workspace across multiple monitors it's hard to imagine that I'd ever prefer web to native.
What I really want is to continue coding on my Mac/PC (a native iPad Pro app would be even better!) and then upload to NodeBox Live when I'm ready to share. The web is great for sharing information, but native still rules for pure creation.
Will I have to choose or can I have both?
John
Support Staff 3 Posted by Frederik De Ble... on 27 Nov, 2015 12:29 PM
Hi John,
This answer is long overdue -- sorry for that!
What you mentioned (work on desktop/ipad, export to web) is certainly an option we're looking into.
Here's my main idea: if we see platforms as a continuum, you could see "the web" on one side, with a huge reach and a programming language (JavaScript) that's available everywhere. It is fast enough for most tasks, but falls short for real-time, interactive graphics and installations (think: Oculus VR, multi-megabyte realtime input streams), and on mobile this speed difference is much more pronounced.
On the other side is C/C++. It is super-fast. It is native to iOS, Mac, Windows, Linux (and Android, using the NDK); and thanks to Emscripten it also runs on the web.
Then there's the stuff in between. Java (which is what NodeBox 3 is written in) is an okay language, it is quite fast, but it's hard to interface with native stuff, or to get running on iOS. Python works very well on servers and in your own projects, but doesn't have a good desktop or mobile story (same for other dynamic programming languages such as Ruby).
Remember, we have a very small team so we don't want to spread ourselves too thin. But I do believe we need these two sides covered: one side that runs on the web (NodeBox Live), and one side that runs where the web is not ideal (phones, Raspberry Pi, physical installations).
So, to answer your question, NodeBox 3 will not disappear, but it will transform into something, an "app" that runs on desktops and smaller devices. I'm not sure if the native web version and the C version will ever be compatible, but we'll provide export options from C (through Emscripten for example).
We've already started development of this C version, and have something running on iPad and desktop. But it's still a long way off. PhD first...
Hopefully that answers your question. Let me know if you have any additional remarks.
F
Support Staff 4 Posted by john on 28 Nov, 2015 09:22 PM
Thank you, Frederick. Very helpful.
I am *delighted* to hear you are exploring a native version for the iPad. I now have an iPad Pro - which might just be the perfect device for NodeBox. Let me know if you need a beta tester!
As for your other ambitious plans I wonder if you are already stretched too thin. Simply maintaining (and improving!) NodeBox 3 across three different platforms is a full time job, let alone NodeBox Live and NodeBox C++. And as I'm sure you know only too well, a PhD can eat years all by itself and drives some people into the loony bin.
I attended the EyeO conference this spring and got a chance to hear and interact with Ben Fry and Casey Reas. They are both proud of Processing but complained that it became an all-consuming chore to maintain. Their solution was to create a foundation to oversee and nurture multiple extensions of Processing (p5.js, Processing.py).
I suspect you are not in a position to snap your fingers and make a foundation appear. But maybe it is time to search for new patrons or write a grant or do something to create a more formal and sustained support system. I worry that without more help NodeBox may evolve too slowly to attract the full audience it deserves.
Keep up the good work and good luck on your PhD!
John
Support Staff 5 Posted by Frederik De Ble... on 10 Dec, 2015 07:19 PM
The foundation is a good idea! We'll discuss with the team.