moda

Modality is an information-centric computing environment. The goals for the project are to improve usability by adhering to a consistent, coherant interaction model, to move to a transparent data/view/function model which will support easy end-user programming, to improve code reuse by building the system around embedded, reused data structures and renderers, and to improve utility by providing automatic, runtime binding of data and relevant tools everywhere in the system. It is currently the project of Erik Pukinskis, but will be open sourced when the basic architecture is in place. More information is available in Erik's wiki.

May 09, 2005 02:00 AM:

After many many hours with essentially zero forward progress, I finally got web browsing working reasonably well about five minutes ago. Well, it works well enough for a screenshot anyway. :)

In the screenshot, you see the currently focused web page. Above it are pages I visited previously, and below it are the pages ahead of it in the history. Clicking a link on a web page adds the new page to the front of the history, rather than just going there.

Here you can see that when I focus my mailbox (the big empty thing with only two messages in it) my documents folder (the pictures in the lower left) and the web pages are pleasantly iconified down below. To pull these back up, you just click on them.

There are still a bunch of issues. Pages in your history are non-interactive and they stay that way even when you click on them. And it's difficult to click on a set of things to focus the whole set instead of focusing just an individual item. I should point out that all of this stuff doesn't care whether it's looking at a web page or a photograph. It's all the same.

These things will be worked out, but I think from here I am going to switch to test driven development. The interfaces are feeling reasonably stable, and things are just getting too complicated to keep going without tests.

Download the snapshot.

May 08, 2005 05:12 AM:

Wow. I am up late. I guess I have been on a roll. Another update:

• The layout algorithm for drawing grids of things is now pretty much correct.
• The HTML viewer is working.
• It captures thumbnails.
• You can click on objects to focus them.

If you focus an object that is part of a set, they are rendered much larger, and the rest of the items in the set are rendered much smaller (see screenshot). I feel very very close to being able to have a browser history that renders old web pages like this as part of the set. All the pieces are in place, I just have to manage when I am using the web browser control, and when I am using a screenshot.

Anyway, I really need to get some sleep. Silly Erik.

Download the most recent snapshot.

May 06, 2005 03:36 PM:

Some exciting progress on Moda:

• The rotate tool is fully functional.
• There is a new translator which can log in to email inboxes and read mail.
• I fixed the flickering problem in the last snapshot.

This screenshot shows a collection of URIs: a link to my email server, and a link to the test folder. The code looks like this:

  Set myLinks = new Set(new GenericType(TypeType.Object, typeof(Uri)));
  myLinks.Add(new Uri("pop://username:password@snowedin.net"));
  myLinks.Add(new Uri(folderBrowserDialog1.SelectedPath, true));
  painter = Factory.GetPainter(myLinks);
  painter.Source = myLinks;

In pseudocode that would be something like this:

  myLinks is a set of URIs (links)
  Add a link to my mail server to myLinks
  Add a link to the test folder to myLinks
  Find me something that knows how to draw sets of things
  Have that thing draw myLinks

It's pretty sweet. Download it here.

May 04, 2005 10:50 PM:

Here's an update of Moda. You'll notice that it handles folders somewhat gracefully (by showing essentially a thumbnail of the folder). And it has no problem with nested folders.

You'll also notice that the information currently under the cursor is highlighted in green, which is fun. And at the top of the currently highlighted thing it says "Rotate". That's a tool, and Moda dynamically looks up what tools are applicable for each bit of information.

Next up is to make that rotate text clickable. Here's the current snapshot.

April 23, 2005 03:18 AM:

Spent about seven hours rewriting Moda from scratch in Windows. Actually, I don't think I used any Windows-specific code, so it should run out of the box on Mono, but I haven't tested it.

Why did I rewrite it? I have been thinking about this architecture in terms of sinks and sources, and translators connecting them. So I wanted to start from that and see where it led me. And actually, I got quite far:

This is a kind of crude folder/image/text viewer, but the back end is quite clean. Auto-discovery of the proper translators is working reasonably well. We do something like this: (obviously this is pseudocode, and wouldn't work.)

  UrlToFile = Find something that will turn uris into files.
  UrlToFile.Source = "/home/erik"
  
  FileToObject = Find something that will turn UrlToFile.Source 
                 into a paintable object.
  FileToObject.Source = UrlToFile.Sink

  Painter = Find a painter for UrlToFile.Sink.

  Painter.Source = UrlToFile.Sink

  this.Paint +=new PaintEventHandler(Painter.Paint)

That's a simplification, but it's literally nine lines of code, so it's not simplified by much. Anyway, I am feeling pretty good about this codebase. It feels closer to correct than other things have in the past. I am using really light interfaces instead of heavier abstract classes... that might be part of it. Another part of it might be that I am just implementing the bare minimum to support the basic use cases I have in mind.

Either way, this is really exciting progress for me, and I don't even feel bad that I didn't get any "important" work done tonight. It is Friday, after all.

Download the snapshot.

January 12, 2005 10:06 PM:

Knocking items off the milestone 2 todo list. Tools are working, though the interface is completely trivial and probably stupid. It basically just specifies an object with a "Run" method. If the object doesn't have a constructor that accepts a ModalityObject, it will just die a flaming death when you try to load it, but that isn't really enforced at the compile level. I really think I need to write a new programming language for Modality. But then I think to myself, Modality will obsolete code anyway, so it doesn't really matter.

Here's a screenshot illustrating dynamic discovery of tools. The example here converts a string to pig latin, and then pirate-speak. I actually wrote the pirate-speak tool, threw it in my plugins folder, and it ran without having to recompile Modality itself. Rock.

I read a little more about Apple's new Automator application. That's a reasonably close approximation to how I think Modality will visualize scripts. It looks like Automator is tool-centered, which is better than being application-centric, but not quite as good as what Modality will be in two respects: 1) it is locked away in a separate application for power users, and 2) it's not data-centered. The idea behind Modality is that we want to change the entire physics of interaction so that people look at their stuff in terms of data that can be manipulated in certain ways. Scripting those manipulations follows naturally from this viewpoint, whereas Automator is an awkward add-on to the application-centered desktop Apple is married to.

Anyway, I'm going to keep cracking at Milestone 2.

January 09, 2005 08:52 PM:

I am somewhere between milestone one and two (see the todo), but I finally got the basic shell working again, so I made a snapshot. It works for me in Mono 1.05.

- Erik

March 25, 2004 11:57 PM:

Niamh mentioned that I haven't been updating my weblog lately. I told her I'll update when I hear from Cornell. No sense in updating if I have no life events to report on.

Well, I still haven't heard from Cornell. I was rejected from UCSD a few days ago, but I don't really want to talk about that. A couple exciting things have happened recently. First, I've been making some progress on my Modality project. It actually handles images, text files and folders moderately intelligently now. A little screenshot:

"Modality draws a folder" or "The most useless file manager ever"

No editing yet, and even the display of folders is pretty sorry, but I'm slowly chipping away at how data is flowing around, which is the whole point of the thing. I have gotten away from XML for various reasons (expanded in my discussion of this project's representation format). The next couple things to tackle are how I'm going to handle composite types, and how interactivity is going to work. I really just want to get to the point where I am trying out new ideas on a stable codebase as I go, and I'm slowly getting there.

So that's what's happening on my geeky side. In real life, I went to see Eternal Sunshine Of A Spotless Mind with Tracy yesterday or the day before. It was really excellent. I caught Jim Carrey on Ellen today, and it just made me want to go back and watch the thing again. In some sense it's a "little things" suspense-romance like Amelie, and it's at least as virtuosic cinemagraphically, but it's a little more thrill, a little less quirk, a little more comedy, and an emphasis on the value of what you already have rather than going out and getting what you always wanted. It's also less idealistic and more true in its portrayal of relationships. I really want to go see it again.

January 30, 2004 02:06 PM:

!!! WARNING !!! GEEK ENTRY AHEAD !!!

I just wrote code that loads a JPEG image into a GTK pixel buffer and then encodes it into an Xml image format of my creation. That's right. XML image format. Fear the disk usage:

         5756 Jan 29 15:09 testimg.jpg
       102687 Jan 30 13:56 testimg.tar.gz
       347428 Jan 30 13:55 testimg.xml

Note that it takes 70 times more space that the original JPEG (20 times more with gz compression.) It also takes something like 10 seconds to do this loading and transformation. In fact, it's a generally accepted fact that XML image formats are stupid, so why am I doing this? Because I'm mad! Mad, I tell you!

October 24, 2003 12:10 AM:

Hot. I've gotten a Batik SVG generating example to run on my computer. Information centric, filesystem->XML->SVG desktop here we come!

cough. This one's optimistic.

July 17, 2003 10:47 PM:

Friends, countrymen, three legged cats and dogs, I give you Ye Ol' First Ever Screenshot of the just-as-ineptly-named-as-the-research-project-that-inspired-it "Modality Source". It doesn't do a damn thing.

The user interface of the future

July 16, 2003 01:33 AM:

Bit of a geeky entry today...

My little project (implementation phase of the unaptly named "modality" project) is coming along pretty well! I have been stumbling along for a good week and a half but I finally feel like I have a decent architectural start for the backend. Right now, the system...

• loads plugins
• takes a URI and tries to find a plugin that understands it
• uses the plugin to get an XML representation of the URI

At the moment there are only two plugins--default and folder--so it still doesn't actually do much, but it's all very exciting nonetheless.

Now that I have the meekest shell of functionality, the next few hurdles are to get unit testing built in, review the code I have to make things more robust, and figure out how embedded documents are supposed to work. Right now, I the plugin that reads folders just creates an XML document with XLinks in it. But should embedded XLinks be transparent to the renderers or should the renderers be responsible for expanding them?

Update: After poking around a bit, I think I have it as it should be for now. Basically, it seems the best way to have a collapsed, expandable XML structure is to have links in the XML itself which can be expanded as necessary. Any other solution would require me to not use the DOM which introduces some complexity I would rather not deal with right now.

It's wierd. In some ways, it's like my brain is too out of shape even to think about recursion.

October 15, 2002 01:22 PM:

This morning I wrote the project proposal for my next independent study. I am very excited about this one, and I want to let that excitement drive my mood if at all possible. I'd rather not dwell on things like the Linear Algebra homework that's due in 40 minutes that won't really be done until tomorrow.

The thrust of the new project is to identify the problems with the current application/document model (disorientation and modality) and then find and implement a solution and compare its usability. I still have a lot of reading to do about quantitative usability testing and I haven't even begun reading about modality and direct manipulation and whatnot, but I am slowly getting that done. The point is--I've got a timeline. The project is moving.