The Web Map Unit of Work

Last week I got a little depressed. I realized my client/stakeholders:

A) Primarily cared about a single dataset

B) Wanted a web page with a map on it

This is easy — it’s mundane. It’s Google Maps’s wet dream — 90% of the users have been brainwashed into thinking the earth is 2, maybe 3 datasets in Platte Carre. It’s a J O B. Whatever features I add to it, putting a single use, single projection, single datatype online still just makes me bored.

GIS has compuational complexities that make it interesting! Take those away… use a single projection, use a single map service, assume no repeat workflows… what do you have? A web map? A web map and some query logic? Perhaps the unit of work for a dull job?

I got into GIS to get away from the monotony of database/forms work. All that work is the same: UI -> validation -> database. Now I’m having visions of a GIS world that had remarkable similarities. Tile Cache + UI -> Map. The reduction in computation complexity necessary to achieve Google Maps style performance has resulted in a similar reduction in complexity of user expectations. Simple and fast is winning the heart and minds of our clients.

And it just might make our jobs suck.

R(D)COM Server NSFW

Having the R stats package available to your web/GIS applications via DCOM sounds pretty sweet.  It does statistics, graphs, and has quite a few GIS related add-ons.

A co-worker was working with a solution which calls R by invoking RTerm.  I stumbled across R(D)COM and thought a DCOM implementation might provide a more reusable solution.  I tried to get him to evaluate/benchmark R(D)COM, but eventually took up the torch myself.

Unfortunately, it seems to be crap. 

 It takes 1 full second to get an R instance.  The least-chatty cross-process IO is barely faster than disk IO.  Cross-process IO hits a scaling wall well before the OS runs out of RAM.  Sending large datasets via the DCOM proxy eats more CPU time than the actual processing within R.  The crowning fault is that R(D)COM fails to destroy one of it’s processes about 5% of the time.  Props to Greg Hill for having way better intuition than me.

Zero Errors { don’t I wish }

I’m porting Geonetwork’s CSW client from Java to .NET.  {I really like Java 1.5+ enums, BTW, very cool.}

I’m not about to brag about my methodology : I try to compile, it complains, I fix where it complained. 

The relief is that I started with 650+ errors, and now I have zero compile-time errors to go.

Tons of runtime errors… but zero compile-time errors!

CSW + GPL = Help!

I’ve been hunting decent Catalogue Service clients.  I want to use one in my web applications, and if I’m going to put any work into it, I’d like to be able to reuse it as an ArcMap plugin.  Now, ESRI’s got their own, closed-source Portal Toolbar.  However, my initial assessment says it has no soul; it relies on ESRI’s metadata shortcut nodes.  Not cool, not extensible, and not useful as a library for my web applications.

I did find a couple of FOSS CSW clients… one from Deegree (LGPL‘ed), and one from GeoNetwork Opensource (GPL‘ed).  To make a long story short, the Deegree one isn’t going to work.  So I’ve strapped on my scuba gear, and begun the descent into the world of the GPL.

Now, the GPL FAQ tells me that I can modify GeoNetwork’s code however I want, as long as I don’t share it.  I’m assuming that means I’m allowed to modify it for my web applications.  Dive much deeper than this, and the barotrauma starts to kick in.  I can’t distributed code that links closed-source libraries (ArcGIS’s) with GPL’ed libraries.  Okay… but where do you draw the line on what’s considered linking? 

Its entirely reasonable to use GeoNetwork’s GPL’ed CSW client to call a closed-source CSW server.  Can I interpret this to mean that an XML web service is enough to differentiate between the GPL’s concepts of bundled programs and combined modules?  So, what, I expose my server-side, modified, GPL’ed CSW client to ArcMap via SOAP?  That doesn’t exactly make me feel happy technology-wise, and violates the spirit of the GPL.

To the point, I’m hoping some OS-GIS people out there will help me do a very bad thing.  I’m hoping GeoNetwork’s CSW client can swim to more pleasant LGPL waters {minus some GPL’ed dependencies}.  I’ve previously made ESRI-kool-aid inspired comments that OS-GIS needs more ArcGIS plug-ins to subvert ESRI users.  I’m not sure GPL’ed clients are going to get us there.

Comments?  Any GeoNetwork people out there?

 UPDATE:

Paul Ramsey says fear not!  The GPL’s code-mixing considerations shouldn’t affect me, see the comments for details.

Tile Cache Musings

OSGeo has a decent amount of information on creating tile caches for WMSs (AKA WMS-Cs).  The general idea seems to be 1) establish a grid system  2) cache any request that conforms to the grid and 3) simply process any other WMS request normally [alternatively deny the request].

The universally accepted problem with caches, then, is that they take up a TON of room.  They’re also usually in a single spatial representation, by virtual of their size.  Hence, if your project requires the ability to reproject, you can’t reasonably build caches for everything.  So instead, you build a tile-cache for the backend and refer to it as overview images, pyramids, or whatnot.  If you’re dealing only with raster data, your WMS’s sole purpose now is to load tiled data, resize and reproject it if necessary, and spit out tiled data. 

 It starts to become clear that WMS-C is something that WMS servers would want to support, in addition to WMS clients.  Doing so would allow custom WMS-C grid definitions and projections, but allow the data owners to maintain only traditional single-projection tile caches.  Native projection WMS-C servers could push tiles to larger banks of WMS-C servers for reprojection and resizing.  The WMS-C tile-owning servers would effectively act as ESRI’s ArcSDE, but exposed via HTTP.

In case people are already doing this, I’ll stick a toe out towards the absurd.  If a standard grid was chosen in a standard projection, this would allow the caching of reprojection-related data.  At minimum, bounding boxes could be stored for each projection, corresponding to the maximum extents fully contained within each native-projection tile.  On the other hand, if you’re going to cache, cache something impressive!!  GDAL’s complex orchestra of affline & reprojection transforms could reasonably(?) be reduced to a serializable set of pixel-pushing operatings without any knowledge of the underlying datasets. 

Stick a whole foot into the absurd, and you’ll see that WMS-C reprojection engines will naturally become RRTWS’s — raster reprojection template web services — the way of the future.  Pixel-pushing thin-clients like Flash will have the ability to reproject without needing bulky clientside APIs.  Client-side vector libraries could use these pixel-lookups to rasterize directly to alternative projections.  Oh, the places we’ll go!

Silverlight first impressions

Being a .NET GIS programmer, a former Flash developer, and an bit of a JavaScript hacker, I’m a natural to be interested in Silverlight.  I remember in 2003 when Microsoft first announced Sparkle (now Silverlight) in the frame of a Flash-killer.  In the mean-time, AJAX seems to have stolen the show.  Silverlight clearly represents a shot over Flash’s bow, but I have to wonder “how does Silverlight compete against AJAX?”

There is really little to say about rendering performance.  Early tests seem to show that WPF/e XAML rendering speed is comperable to Flash and DHTML.  Pretty good performance for a beta product.  Overall, WPF/e should work great for displaying small GIS vector feature datasets.  WPF/e is nicely integrated, and its straightforward how to programmatically create layout.

The part of Silverlight that interests me the most is its support for a subset of .NET.  Microsoft has a demo showing .NET for Silverlight running up to 1000x faster than Javascript, effectively 10-15x faster than Flash’s Actionscript.  Sweet.  In an odd coup, the DLR will allow you to write your Silverlight .NET code in Javascript, facilitating high-performance AJAX ports.  Beyond that, a quick look at the HTML bridge quickstarts shows that we need not rely on WPF/e XAML – Silverlight can bind to the HTML DOM quite handily.   Again, score for Silverlight. 

Here is where my opinion really starts to diverge.  I can’t find Silverlight’s supposed “serialization classes for handling data.” I haven’t seen them in the remoting quickstarts or a quick API examination.  Yes, there’s support for plain XML, REST, and SOAP… but I can’t find any automatic object/class-level serialization.  Until I see object serialization between ASP.NET and Silverlight, its not the JSON-killer I hoped for.

In fact, despite the high-technology, its hard to see what Microsoft has done here.  Certainly they’ve given their C# programmers a skyhook into browser-based development.  They’ve upped the computing-performance ante for browser capabilities.  That a pretty big deal.  However, Silverlight still comes off as direct competition against Flash, not a competitor to AJAX.  So why push out a zero-installation-base plugin when you own the browser marketshare?  Had IE7 matched Mozilla’s support for SVG/Canvas, E4X, and Javascript getters/setters — this would have given us 80% of the functionality of Silverlight’s limited APIs.  Perhaps because they’ve been planning this since 2003. 

 UPDATE:

As pointed out in the comments, Silverlight is based on AJAX in a very non-disruptive way.  The afforementioned support for SOAP is really support for ASP.NET AJAX JSON-enabled web services.  Beyond that, this is all documented exactly where I said it wasn’t … http://silverlight.net/QuickStarts/Remote/UsingJSON.aspx.  …And I wasn’t even trying to troll.

ESRI and Silverlight

ESRI is hiring for a “.NET software development programmer” to “build core business logic for our Web-based GIS applications” with “Background in graphics programming and WPF” and “Familiarity with .NET 3.0″.  They want someone with “Experience with ASP.NET AJAX” but who can “master new technology quickly” in a “changing environment.”

 Can anyone say Silverlight?