Persisting Model State with JSON

I’m departing from my normal show a lot of code methods here, to talk about an idea.

ASP.NET MVC denegrates ASP.NET Web Form’s old state-maintenance construct called Viewstate.  Without Viewstate, many developers are wondering how — more aptly where — to preserve the state of a web-application.  Here I will be explaining my favored answer — serializing your model to JSON — and showing why.

SOME HISTORY

In ASP.NET Web Forms, page state was stored within a highly compressed, encrypted hidden form field.  This allowed .NET web developers to persist the state of the GUI controls with little-to-no effort.  Unfortunately, AJAX came along.  AJAX developers found ASP.NET Web Forms didn’t play nicely at all; and that since Viewstate was encoded, AJAX developers couldn’t easily work-around ASP.NET Web Forms limitations.

ASP.NET MVC

The Model-View-Controller architectural pattern could easily be called the state/GUI/event-handler pattern.  Personally, I feel that most ASP.NET MVC examples nail showing off the Controller, and by proxy do a pretty alright job showing off the View.  But simple examples often have simple domain logic, and hence the Model and state-maintenance has been largely ignored.  This is important to note, because HTTP is a stateless protocol, yet the MVC pattern was designed for persistantly stated desktop applications.

“The model manages the behavior and data of the application domain, responds to requests for information about its state (usually from the view), and responds to instructions to change state (usually from the controller).” – Steve Burbeck, from the paper first describing MVC

STATE MAINTANENCE IN MVC

Given the statesless nature of HTTP, most ASP.NET MVC examples imply that the Model is little more than a “page model” with some supporting domain logic.  For the Model to act across multiple pages, we must persist that model between page requests… and we have options… databases, mutiple hidden form fields, sessions, cookies, etc..

The common denominator is that each page request must somehow reconnect to its Model.  In many cases, the Model — or at least the current pertinant chunk of the Model — is small enough to reasonably be persisted with each page, much as Viewstate was.   The huge advantage to custom Model persistance  comes in saving the Model to a hidden-form-field in the form of Javascript Object Notation, or JSON.  Using JSON means that not only will the Model be persisted, it will also be available for AJAX developers to update without full-page-postback.

ISN’T PUTTING THE MODEL IN THE VIEW BAD?

Considering the alternatives, no.  Sure, persisting the Model to a database would have some security benefits — at the expense of connection overhead.  Sessions or cookies work as well — with their own known limitations.  You could also — God forbid — break your model into little bits and persist each one as necessary.   Let me say this is the WORST possible thing you could do, because it requires the View to know the entire Model to do its job.  The code to serialize to JSON is the same for any object you throw at it, and changes to the Model won’t have to be duplicated in each View.

CONCLUSION

Of course, there are limitations to persisting your Model via JSON.  HTTP Get/Post have character limitations you must consider.  You will want to validate your model on each page post.  Overall, however, the benefit to AJAX developers outweighs the security awareness necessary to implement this state-management architecture, and prevents your ASP.MVC architectures from falling into the rut of “page-state”.

Fast unknown method invocation for loose coupling

I was recently tasked to refactor a WinForms application. Now familiar with ASP.NET MVC, I wondered how easy it would be to apply existing ASP.NET MVC constructs to WinForms.  Implementing an IController is easy enough, but there’s still the matter of loose coupling within the IController’s execute method.

I’ve previously shown using reflection to allow loosely-coupled method invocation in ArcGIS Server Object Extensions, but reflection is wicked slow.  I knew that people were writing ORM binding using IL op-codes, and I figured that Microsoft had done something similar for ASP.NET MVC.

It turns out that LINQ expressions can be compiled into IL op-codes and that people are doing it to invoke unknown code quickly, without reflection.  No need to fire up Reflector to see Microsoft’s worth seeing LINQ implemenation, you can simply check CodePlex.

Using ProjNET for point reprojection

Are you – yes, you– firing up ArcObjects code just to do simple point reprojection?  Check out Proj.NET.  Grab the EPSG-code reader and point projection code from the very helpful FAQs and you’ll get something like this: [edit: fixed, thanks Morten ... definitely read the comments, this code was meant to show how easy things could be... its definitely not the ProjNet best practices]

public static double[] projectFromWgs84(double X, double Y, int destinationEpsgCode)
{
   double[] input = new double[] { X, Y };
   SC.ICoordinateSystem destProj = SridReader.GetCSbyID(destinationEpsgCode);
   SC.ICoordinateSystem sourceProj = SridReader.GetCSbyID(4236);
   SCT.CoordinateTransformationFactory ctf = new SCT.CoordinateTransformationFactory();
   SCT.IMathTransform xForm = ctf.CreateFromCoordinateSystems(sourceProj, destProj).MathTransform;
   return xForm.Transform(input);
}

ASP.NET MVC + Ext + WCF

Dave Bouwman wrote up a nice post about using MVC with AJAX.  Dave’s example, however, requires manual serialization / deserialization.  Fortunately, Omar Al Zabir wrote a filter that leverages WCF’s DataContractJsonSerializer to accomplish just this. 

I’ll spare the code duplication, but here’s how to call it from Ext:

 Ext.Ajax.request({
    url: ‘/myUrl.mvc/testDTO’,
    headers: { ‘Content-Type’: ‘application/json’ },
    success: mySuccessFunction,
    failure: myFailureFunction,
    jsonData: myJavascriptObject
});

Using the ArcSDE Java API in .NET

I’m a .NET guy.  I have no desire to fool around with ArcSDE’s C API’s… building them, COM interop, whatever the heck would be required.  I’m also definitely reluctant to call SDE’s command-line tools from code.

Enter IKVM.  “IKVM.NET is an implementation of Java for Mono and the Microsoft .NET Framework.”  “IKVM.NET includes ikvmc, a Java bytecode to .NET IL translator.”  Run the ArcSDE Java API jars through ikmc, and you’ve got yourself a .NET ArcSDE library:

ikvmc jpe_sdk.jar -target:library
ikvmc icu4j_3_2.jar -target:library
ikvmc jsde_sdk.jar -target:library -r:jpe_sdk.dll -r:icu4j_3_2.dll
ikvmc concurrent.jar -target:library

The above .BAT file will generate 4 ArcSDE .NET dlls; giving you access to the Java API from .NET.  You also need to include the IKVM dlls in your project.  You loose some of the intellisense and error reporting goodness, but a quick test makes it look like all is working. :c)

AGS + SDE – MXD + SOE

This little bit of ArcGIS Server Object Extension code can take an ArcGIS Service based on an empty MXD, and dynamically add all the feature classes from an SDE instance to it.  Its one of my first SOEs, but it seems to run just fine. Somewhat unique about this code is that this SOE has no callable methods, its mere presence is its power.

 

using System;
using System.Runtime.InteropServices;
using System.EnterpriseServices;
using ESRI.ArcGIS.Carto;
using ESRI.ArcGIS.esriSystem;
using ESRI.ArcGIS.Geodatabase;
using ESRI.ArcGIS.Server;

namespace com.wordpress.mapwrecker
{
    [AutomationProxy(true), ClassInterface(ClassInterfaceType.None), GuidAttribute("a9ed9e17-d2b2-461c-9da3-2915af9a3f00")]
    public class SDE_Eater : ServicedComponent, IServerObjectExtension, ILogSupport
    {
        private ESRI.ArcGIS.esriSystem.ILog2 m_log;

        private object getSingleton(string progId)
        {
            return Activator.CreateInstance(Type.GetTypeFromProgID(progId));
        }
        public void addAllSdeLayersService(IServerObjectHelper m_SOH)
        {
            IWorkspaceFactory sdeWorkFact = null;
            try
            {
                MapServer ms = m_SOH.ServerObject as MapServer;
                IMapServerObjects mso = ms as IMapServerObjects;
                IMap map = (IMap)mso.get_Map(ms.DefaultMapName);

                sdeWorkFact = (IWorkspaceFactory)getSingleton("esriDataSourcesGDB.SdeWorkspaceFactory");
                IPropertySet propertySet = getSdeProperties();
                IWorkspace pFWorkspace = sdeWorkFact.Open(propertySet, 0);
                IEnumDataset penumDataset = pFWorkspace.get_Datasets(esriDatasetType.esriDTFeatureClass);

                IDataset pDataset = penumDataset.Next();
                while (pDataset != null)
                {
                    IFeatureLayer featureLayer = new FeatureLayer();
                    featureLayer.FeatureClass = (IFeatureClass)pDataset;
                    m_log.AddMessage(3, 110000, "SDE SOE found: " + pDataset.Name);
                    featureLayer.Name = pDataset.Name;
                    map.AddLayer(featureLayer);
                    pDataset = penumDataset.Next();
                }
                mso.RefreshServerObjects();

            } catch (Exception e) {
                m_log.AddMessage(1, 110000, "SDE SOE error: " + e.StackTrace);
            } finally {
                int refsLeft = 0;
                do { refsLeft = Marshal.ReleaseComObject(sdeWorkFact); } while (refsLeft > 0);
            }
        }

        private IPropertySet getSdeProperties()
        {
            IPropertySet propertySet = new PropertySet();
            propertySet.SetProperty("SERVER", "blah");
            propertySet.SetProperty("INSTANCE", "sde:sqlserver:blah");
            propertySet.SetProperty("DATABASE", "SDE");
            propertySet.SetProperty("USER", "blah");
            propertySet.SetProperty("PASSWORD", "blah");
            propertySet.SetProperty("VERSION", "sde.DEFAULT");
            propertySet.SetProperty("AUTHENTICATION_MODE", "DBMS");
            return propertySet;
        }

        public void Init(IServerObjectHelper pSOH)
        {
            m_log.AddMessage(2, 110000, "SDE SOE is loaded");
            addAllSdeLayersService(pSOH);
        }

        public void Shutdown()
        {
            m_log = null;
        }

        public void InitLogging(ILog Log)
        {
            m_log = (ESRI.ArcGIS.esriSystem.ILog2)Log;
        }
    }
}

Getting “localhost.” to work with IIS7

I’m not 100% sure why, but with IIS7, I am unable to get connect to my local IIS instance using Fiddler’s magic “localhost.” hostname.  Normally this hostname allows localhost traffic to be monitored within Fiddler. Out of the box, IIS7 was returning me the following error on requests to “localhost.”:

Bad Request – Invalid Hostname

Trying to add a secondary binding with “localhost.” explicitly set as the host yeilds another error, this time from the IIS Management Console:

Value does not fall within the expect range.

I found that editing Fiddler’s CustomRules.js provides a quick workaround while still allowing you to use the exact “localhost.” name.  Simply add the following line to the “onBeforeRequest” function:

if (oSession.host.ToUpper() == “LOCALHOST.”) { oSession.host = “localhost”; }

This fix of course only works when Fiddler is running, so please comment if you know of a workaround within IIS7.  (Alternatively, please comment if you have no issue in IIS7 — that my box simply has a case of the Vistas.)

ArcGIS Server Virtual Earth Tile Server vs. ArcIMS

Time to change the name Dave.

     

using System;
using System.Text;
using ArcDeveloper.TileServer.Interfaces;
using System.Net;
using System.IO;
using System.Xml;
using System.Drawing;
using System.Drawing.Imaging;   

namespace ArcDeveloper.TileServer.ArcIMS
{
    public class ArcImsTileProvider : ITileProvider
    {
        ///
        /// Private member containing the web service url
        ///
        private string _webServiceUrl = "";   

        ///
        /// Constructor that takes a web service Url
        ///
        ///
        public ArcImsTileProvider(string webServiceUrl)
        {
            _webServiceUrl = webServiceUrl;
        }   

        ///
        /// Get the tile image
        ///
        ///
        ///
        ///
        /// Tile data stream
        public System.IO.MemoryStream GetTile(ITileExtent extent, int tileHeight, int tileWidth)
        {
            MemoryStream returnStream = null;   

            try
            {
                string axl = getAXL(extent, tileHeight, tileWidth);
                Stream responseStream = PostData(_webServiceUrl, axl);
                XmlDocument xd = new XmlDocument();
                xd.Load(responseStream);
                string imgURL = xd.GetElementsByTagName("OUTPUT")[0].Attributes["url"].Value;
                responseStream.Close();   

                Bitmap bm = new Bitmap(new WebClient().OpenRead(imgURL));
                returnStream = new MemoryStream();
                bm.Save(returnStream, ImageFormat.Png);
            }
            catch(Exception e)
            {
                Bitmap bm = GetMessageTile(e.Message);
                returnStream = new MemoryStream();
                bm.Save(returnStream, ImageFormat.Png);
            }   

            return returnStream;
        }   

        ///
        /// Gets the ArcIMS GetImage AXL request.
        ///
        ///
        ///
        ///
        /// GetImage ArcIMS AXL request
        private string getAXL(ITileExtent extent, int tileHeight, int tileWidth)
        {
            StringBuilder sb = new StringBuilder();
            sb.Append("<?xml version=\"1.0\" encoding=\"UTF-8\" ?>");
            sb.Append("<ARCXML version=\"1.1\">");
            sb.Append("<REQUEST>");
            sb.Append("<GET_IMAGE>");
            sb.Append("<PROPERTIES>");
            sb.Append("<ENVELOPE minx=\"" + extent.lon2 + "\" miny=\"" + extent.lat2 + "\" maxx=\"" + extent.lon + "\" maxy=\"" + extent.lat + "\" />");
            sb.Append("<IMAGESIZE height=\"" + tileHeight + "\" width=\"" + tileWidth + "\" />");
            sb.Append("</PROPERTIES>");
            sb.Append("</GET_IMAGE>");
            sb.Append("</REQUEST>");
            sb.Append("</ARCXML>");
            return sb.ToString();
        }   

        ///
        /// Gets a tile with an error message burned into it.
        ///
        /// The message.
        /// An error message PNG
        private Bitmap GetMessageTile(string message)
        {
            Bitmap tile = new Bitmap(Properties.Resources.blankTile);
            Graphics graphics = Graphics.FromImage(tile);
            Font drawFont = new Font("Arial", 12);
            SolidBrush drawBrush = new SolidBrush(System.Drawing.Color.Black);
            SizeF len = graphics.MeasureString(message, drawFont);
            graphics.DrawString(message, drawFont, drawBrush, new PointF(10, 10));
            graphics.Dispose();
            return tile;
        }   

        ///
        /// Does an HTTP Post, returning a stream.
        ///
        /// The URL to post to
        /// The data to post
        /// The Http response stream
        private Stream PostData(string sURL, string postData)
        {
            HttpWebRequest webReq = null;
            HttpWebResponse response = null;
            Stream requestStream = null;
            StreamWriter writer = null;
            Stream responseStream = null;
            try
            {
                webReq = (HttpWebRequest)WebRequest.Create(sURL);
                webReq.Method = "POST";
                webReq.ContentType = "application/x-www-form-urlencoded";
                webReq.Timeout = 0xea60;
                webReq.AllowAutoRedirect = true;
                requestStream = webReq.GetRequestStream();
                writer = new StreamWriter(requestStream);
                writer.Write(postData);
                writer.Flush();
                writer.Close();
                requestStream.Close();
                response = (HttpWebResponse)webReq.GetResponse();
                responseStream = ((HttpWebResponse)webReq.GetResponse()).GetResponseStream();
            }
            finally
            {
                if (writer != null) writer.Close();
            }
            return responseStream;
        }   

    }
}   

Winners: Python-fearing .NET folks still using ArcIMS.  All six of you!

Losers: “ArcGIS Server Virtual Earth Tile Server” as a name.  Die.

LINQ vs LINQ

I’m just getting into LINQ, and experimenting with what goes on behind the scenes.  LINQ uses the idea of a “DataContext” as an interface to the database, much like the older “DataSet” acts to encapsulate multiple table structures.   One notably lacking feature of LINQ’s database-derived classes is the ability to traverse many-to-many relationships without touching the intermediate cross-reference table. 

Assuming an existing “vsNetwork” entity, you can query a many-to-many relationship directly against the ”DataContext” ala:

from f in myDataContext.ScienceFocus
join vsf in vsDc.VitalSignFocus on f.FocusId equals vsf.FocusIdf
where vsf.VitalSignIdf == vsNetwork.VitalSignId
select f;

Again assuming an existing “vsNetwork” entity, you can also query by walking the database-derived class’s table-relations, ala:

vsNetwork.FXrefVitalSignFocusList.Select(vsf => vsf.FTluScienceFocus).ToList();

The magic of LINQ exists within all things that implement the IQueriable interface.  This allows the keywords “select”, “where”, “join”, etc. to be applied to a collection of objects.  Note, however, that there are differences in how “IQueriable” is implemented.  The examples above, indeed generate markedly different SQL.  Respectively,

SELECT [t0].* FROM [tlu_Science_Focus] AS [t0] INNER JOIN [xref_Vital_Sign_Focus] AS [t1] ON [t0].[Focus_ID] = [t1].[Focus_IDF] WHERE [t1].[Vital_Sign_IDF] = vsNetwork.Vital_Sign_ID

versus

SELECT [Focus_IDF] FROM [xref_Vital_Sign_Focus]  WHERE [Vital_Sign_IDF] = vsNetwork.Vital_Sign_ID

*For Each Focus_IDF…*

SELECT * FROM [tlu_Science_Focus] WHERE [Focus_ID] = Focus_IDF

We see that the latter runs multiple queries instead of using a SQL join.  The source of this inefficiency is described by The Wayward Weblog:

“[EntitySet's] ToQueryable() wraps your IEnumerable<T> in IQueryable<T> clothing, uses the Queryable infrastructure to let you build up your own expression tree queries, and then when you enumerate it, the expression is rebound to refer to your IEnumerable<T> directly”

In this case, “vsNetwork.FXrefVitalSignFocusList” is of type EntitySet, which does not implement IQueriable directly, but rather through IEnumerable.  So not only does this shorter syntax lose efficiency by issuing multiple and superfluous queries, it also loses efficiency by iterating through each xref table in memory.  In the case of large sets, using LINQ against IEnumerables can be an issue.  It has been enough of an issue that people are adding indexing for in-memory LINQ objects, rather than simply iterating collections.

In earlier Visual Studio 2008 CTP’s, LINQ EntitySets implemented the iQueriable interface (directly?).  I’m not sure if at that point, this was implemented to elegantly generate SQL, I simply know that O’Reilly’s C# in a Nutshell author Joseph Albahari made some fuss about the change. 

It appears that Mr. Albahari is behind a library called LINQKit designed to mitigate his qualms with the LINQ’s recently-neutered EntitySet classes.  While this library appears interesting, it critically doesn’t make EntitySets natively IQueriable.  Hence, I don’t see a way where LINQKit allows effortless, short syntax, efficient SQL queries to be built from existing database-derived classes.

Winners: Tautologically, LINQ totally won this battle.

Losers: LINQ EntitySets, for failing to combine short syntax and efficient SQL generation.