User-Centric Development

The keynote at DevLearn 2008 this past past fall was given by Dan Roam, the author of The Back of the Napkin. Dan spoke about the importance of thinking visually, especially when problem solving.

His presentation spurred a conversation between myself and the other Lampo attendees at the conference: Jon Shearer and Michael Finney. We were discussing how bad things can happen when software engineers find a neat “feature” they can work into an app, regardless of whether the users want or need it.

This brought Finney to a perfect, real-life example of this concept in action. He had recently purchased a new Gateway laptop with a built-in 802.11n wireless network adapter. When using the wireless, however, he had noticed that it would occasionally slow to a crawl. It’s never terribly convenient to troubleshoot this type of problem, especially when so many components are involved (ISP, broadband modem, wireless router, laptop hardware, OS…) and the problem is intermittent.

Well, last weekend, he had a guest in his house with a laptop, and he was able to do a side-by-side speed test during one of these episodes, and he discovered that while the other laptop was getting about 20mbps down, his was getting about 20kbps down.

This took the network out of the troubleshooting stack, so he really dug into the laptop to locate the source of the problem. Before long, he found a “feature” that is new to Windows Vista.

As with prior versions of Windows, you can create and customize power schemes to use when on battery power in order to preserve battery life by throttling the power that certain components use. It is common to scale back the processor speed or the LCD brightness to achieve longer batter life.

Well, Vista has added Wireless Network Adapter Throttling into the default power saver scheme, effectively crippling your network speeds in the name of battery preservation. Let’s think this through for a minute. I want to preserve battery life, so having to sit and wait 1000x longer for a website to come up (all while my LCD and processor burn through battery while I sit and do nothing) is the answer?

Just because you can do something as a developer with the technology you have, doesn’t mean you should.

And just in case you got to this post by searching for an answer to your network slowdown woes, here’s how you change this setting in Vista. Control Panel -> Hardware & Sound -> Power Settings -> Change Plan Settings (on your currently selected power plan) -> Change advanced power settings -> Wireless Adapter Settings -> Power Saving Mode. That’s right, 7 clicks deep, one of which includes clicking on “Sound”. *sigh*

The Sweet Java Topology Suite – Part II

In a previous post, we described how we started using the Java Topology Suite (JTS) to manipulate postal/zip code polygons that we are viewing in an application built on MapQuest’s Flex API. Since then, we have added the ability to join multiple postal codes into territories. Sometimes over 1,000 postal code polygons will be combined to form a single territory.

We ran into two significant technical hurdles. First, MapQuest’s API doesn’t support polygons with inner holes. So, a donut-shaped polygon would just look like a circle, with no hole in the middle. The other problem was that some of the postal codes were so complicated that the unify process would fail.

Union of postal code polygons with a hole in the middle

Union of postal code polygons, missing the hole in the middle

If you read the other article, you saw that we did use JTS to simplify polygons (by reducing the number of points that make up the polygon). However, we didn’t end up using those in production because the edges of the simplified polygons would not line up. They end up looking like broken glass, because the simplify process had no regard for adjacent polygon edges.

Simplified polygons with edges that don't line up

So, we set out on an adventure to simplify the polygons so that the edges of the simplified postal codes matched up. We received some very responsive and helpful guidance from Martin Davis, one of the principle developers of JTS. He also pointed us to the open source tool OpenJUMP, which he also helped to build. Source code from that tool was very helpful as we created our own automated simplification process.

Here’s the simplification process in a nutshell:

1. Convert the MapQuest postal code polygon data for the current patch (like the lower 48 states) to Well-Known Text (WKT) and save each postal code polygon to an individual file on the file system. For the lower 48 states, this resulted in more than 41,000 files. Here is an unsimplified version of the few polygons we’ll simplify in this example:
   

   Original, unsimplified postal code polygons

2. Read all of the WKT files, one per postal code, and store them as JTS Geometry objects in a collection. To support step six (below), we store the postal code in the geometry object using the very handy Geometry.userData property. That way, each original/source geometry remembers what postal code it represents.
3. Use JTS to convert the polygons to merged LineString objects. This creates a collection of the outlines of every polygon, where the common polygon edges become a single line.
   

   Extracted border lines of original polygons

4. Use JTS to simplify the merged LineStrings by reducing the number of coordinates that define each line. Our code iterates across every merged LineString and uses JTS’s DouglasPeuckerSimplifier with a simplify tolerance of 0.01.
   

   Simplified polygon border lines

5. Use JTS to create polygons from the simplified LineStrings. The primary JTS class was the magic Polygonizer class, along with code from OpenJUMP that prepared the line data for the Polygonizer.
   

   New polygons made from simplified lines

6. Now the tough part. We have a collection of simplified polygons, but they aren’t linked to any postal codes, so we can’t find the polygon and use it in our application. We needed to match the simplified polygon with the original. Since this is among the most involved processes, I’ll describe it in a bit more detail:
   1. Add each of the original polygons to a JTS SpatialIndex called STRtree. The STRtree provides a quick query interface to find polygons that fall within a spatial constraint.
   2. Iterate through each of the simplified polygons, and:
      1. Query the STRtree to find all of the original polygons that touch the envelope (bounding rectangle) of the current simplified polygon.
      2. Find the polygon in that set which has the smallest distance between its center point and the simplified polygon’s center point.
      3. Once the best matching simplified polygon is found, we copy the postal code from the original Geometry’s userData.
      4. Some simplified polygons have no match in the original set because of holes, so those non-matches are thrown out in this process.
   3. Now that each simplified polygon has been identified as matching a postal code, we write new WKT files for each postal code. Our code that writes these files automatically creates MultiPolygon objects for those postal codes that are made up of more than one polygon.
   

   Simple polygons that remain after match with originals

In order to run this process on the lower 48 United States, I had to allocate 7GB of my 8GB of RAM to the JVM so that all 41,000 polygons could be simplified at the same time. Fortunately, it’s worth the time to build. Here are the number of coordinates needed to represent all of the polygons for the three areas, both originally and after simplification, along with the savings realized:

	Coordinate Count
	Original	Simplified	Reduction
Lower 48 United States	6,276,000	544,000	12x smaller
Alaska	262,000	15,000	17x smaller
Hawaii	72,000	960	75x smaller

Here’s a larger area of polygons, before and after simplification:

Original postal code polygon sample

Simplified postal code polygon sample

In order to create polygons that maintain any holes in the middle with MapQuest’s polygon API, we used JTS to cut a small slice between any inner features and the exterior of the polygon. This leaves a line in the middle of the polygon, but it’s more acceptable than no hole at all. Hopefully MapQuest will support polygons with inner holes in a later release. In fact, it would be really cool if MapQuest would incorporate other structures and features from JTS, including native WKT support.

Simplified postal codes on map

Territory on map with hole enabled by slice

We are very grateful for the Java Topology Suite and the polygon processing it allowed us to complete. The project we’re building for Dave’s Endorsed Local Provider program will be much more successful with these improvements.

How Much Memory Does This Take?

Ever wonder how much memory an object takes?

The web has lots of  great theoretical answers to that question.

Theories are great, and they give lots of great insights into how a Java compiler works. But what if things aren’t perfectly clean-cut? There are plenty of reasons to want proof including the following possibilities:

• Say you use the Spring Framework or some other mechanism where your objects aren’t composed until runtime
• Maybe you use a library/jar/package/swf/swc/etc, you don’t have access to the underlying code, and you’re trying to weigh its memory footprint
• Maybe your objects are composed differently in different states
• Maybe you use a Rapid Application Development language such as  ColdFusion or PHP and want to see how much overhead they bring along
• Maybe you’re just a hard-core detailed person who wants to prove that the theory really matches reality

We’ve had to troubleshoot ColdFusion apps for memory usage, and we’ve found this approach to be helpful:

1. Call for garbage collection (if applicable)
2. Take a snapshot of how much memory is used
3. Instantiate a whole bunch of copies of the class in question, and stuff them into an array
4. Call for another garbage collection (if applicable)
5. Take another snapshot of how much memory is used
6. Subtract the first memory reading from the second, divide by the number of objects created, and you have a pretty good approximation of how much memory each object takes

As a side-note, the more objects you create in your test, the more your test will drown out background noise from other activity, and the more accurate your reading will be on each object.

After researching this method, we honed it a bit and made some useful discoveries. For example, in ColdFusion 7, each CFC instantiated takes up ~2kB. On top of that, each <cffunction></cffunction> in that cfc takes up an additional ~150 bytes.  So the abc.cfc “class” (found below) with 3 empty methods takes up ~2.3kB. (disclaimer: I’ve heard but not yet personally verified that memory usage improves in CF8 and beyond.)

In our scenario, we had CFCs inheriting a data persistence layer including about 50 methods, our CFCs included about 50 more in addition to member data in the variables scope. Without fully understanding the under-the-hood functionality of ColdFusion 7, we were instantiating 30 CFCs per user at 20kB each (~half a meg) for a few thousand users. Then we wondered why we kept eating through our available memory so quickly.

After making this discovery we spent a few days re-factoring some code hoping to use resources better. We limited the inheritance of the persistence layer to where it was necessary. We re-factored the code to use ColdFusion queries rather than CFCs (queries wound up being MUCH lighter than similar CFCs). When we launched the changes our application used about 40% less memory.

As hinted earlier, this algorithm can also be used to analyze memory usage by basic Java components. One question I’ve dabbled with is how much memory java code takes. I may cover that subject in another post someday, but for now I’ll leave that as an exercise for the reader. In the meantime, here’s a ColdFusion script that implements the algorithm described above.  Enjoy!

<!---this part executes in a cfm--->

<cfset runtime = CreateObject("java","java.lang.Runtime").getRuntime() />
<cfset myArray = arrayNew(1) />
<cfset intNumberOfObjectsCreatedPerLoop = 10 />
<cfset intTotalNumberOfObjectsCreated = 0 />
<cfset intX = 0 />
<cfset intBytesBeforeTest = 0 />
<cfset intBytesAfterTest = 0 />
<cfset intBytesCreated = 0 />
<cfset intBytesIn1MB = 1048576 />
<cfset memoryThresholdInMB = 50/>
<cfset obj = "" />

<cfset runtime.gc() />
<cfset intBytesBeforeTest = runtime.totalMemory() - runtime.freeMemory() />
<cfloop condition="intBytesCreated lt (intBytesIn1MB * memoryThresholdInMB)">
<cfset intTotalNumberOfObjectsCreated = intTotalNumberOfObjectsCreated +
		intNumberOfObjectsCreatedPerLoop />
	<cfloop from="1" to="#intNumberOfObjectsCreatedPerLoop#" index="intX">
		<cfset obj = CreateObject("component", "abc") />
		<cfset arrayAppend(myArray, obj) />
	</cfloop>
	<cfset intBytesAfterTest = runtime.totalMemory() - runtime.freeMemory() />
	<cfset intBytesCreated = intBytesAfterTest - intBytesBeforeTest />
</cfloop>
<cfoutput>
<cfset runtime.gc() />
#intBytesCreated / 1024# kilobytes of objects created.<br>
Each CFC takes approximately :
#(intBytesAfterTest - intBytesBeforeTest) / intTotalNumberOfObjectsCreated / 1024#kB
</cfoutput>
<!--- this part resides in a separate file (abc.cfc) in the same directory --->

<cfcomponent name="abc">
	<cffunction name="a" output="false"><!--- Really important code here. ---></cffunction>
	<cffunction name="b" output="false"><!--- Really important code here. ---></cffunction>
	<cffunction name="c" output="false"><!--- Really important code here. ---></cffunction>
</cfcomponent>

The Hits Just Keep On Coming

It’s been an interesting year for us webmonkeys.

In the last several months a few big banks, insurance companies, and/or financial companies were deeper into sub-prime mortgages than they should have been. The US government stepped in to either loan money (AIG) or take over (Fannie Mae, Freddie Mac) said organizations. The aforementioned steps and those that followed have since blown open the door for much more intervention.

If you’re very familiar with Dave Ramsey, you probably know that we don’t like debt very much around here. To try and stop the one of the first multi-billion dollar bailouts from becoming law, Dave went on 32 different radio and tv shows within 36 hours talking about The Common Sense Fix as an alternative to buying hundreds of billions of bad debts.

This all came together very quickly. Needless to say, it left us in the exciting position of being able to serve more people than our website was comfortably equipped to handle at the time.

At that point in time, many of the pages on our website are generated dynamically every time someone visits. We quickly took static “snapshots” of the high-traffic dynamic pages and put them out there to speed load times. In addition, we moved a lot of our images, .css, .js, and other static files out to be served by Akamai instead of by our local servers. It’s a bit more difficult to update things when Akamai hosts them, but they have a ginormous amount of capacity in their network. To use another example from the industry, Yahoo uses Akamai for most of their static content as well. We were pleasantly surprised by how dramatic of a speed increase came from using them. Perhaps even more exciting, Akamai’s bandwidth price is less expensive than our more traditional hosting arrangements.

What else did we do to prepare? In addition to upgrading some hardware, we also looked at our code to see how we could reduce database usage. We wrote this query to analyze the top 50 queries performing IO on our database:

select top 50
qs.total_worker_time / execution_count as avg_worker_time,
substring(st.text, (qs.statement_start_offset/2)+1, ((
case qs.statement_end_offset
when -1 then datalength(st.text)
else qs.statement_end_offset
end – qs.statement_start_offset)/2) + 1) as statement_text,*
FROM sys.dm_exec_query_stats as qs with (nolock)
cross apply sys.dm_exec_sql_text(qs.sql_handle) as st
order BY avg_worker_time desc

As a quick disclaimer, the query above takes a cross section of the I/O reads executed over the last 10 minutes. We did notice significant differences in the pie charts generated at different times of the day. If you’re going to be investing a lot of time or money into making optimizations, you need to run several of these reports at different times and do some experimentation of your own to see what makes the most sense for you.

We took the results of the above query, we grouped the queries by application, and we made a quick excel pie chart like this one:

Database Usage by Application

With this data, we were then able to dig down into the code and analyze which activities were causing the heaviest database load. In a pinch we were able to quickly shut off a scheduled task and optimize another heavy piece of code to reduce our database IO by 50%.

After the first round of optimizations, we ran the report again and created reality-based plans for further optimization should it be needed.

Getting actual concrete data on the queries actually causing I/O reads was empowering. At the outset of this we brainstormed ways to increase capacity. We proposed some fairly involved projects to optimize a few parts of the site that we knew executed queries. Thankfully we waited for real data before acting, because none of the ideas we initially proposed would have fixed any of the 50 slowest queries.

We made it through this event without serious disruption to our web visitors. However, the experience is really making us think twice about what would happen if we were ever to be Dugg or Slashdotted in a big way.

What an INCREDIBLE problem to have!

More ColdFusion Testing Woes With Java Objects

Once again, my team is currently working on a Flex-based mapping application and I’m working on the back-end using ColdFusion to grab the data from our SQL server, package it in a Java based data transfer object (DTO), and send it to the requesting Flex app.

Another minor glitch I ran into while trying to test the ColdFusion method calls is with the SerializeJSON function built into ColdFusion 8. Because of the way BlazeDS works, in order to send objects from the Flex app back to ColdFusion we have to convert the object to a JSON string, send it, and then rebuild the object in ColdFusion from the JSON string. My teammate explains the process in his blog entry about the tech stack “Flex, ColdFusion, Java, and BlazeDS: with JSON?”

The issue is simple. I want to test methods in the ColdFusion service we are writing in ColdFusion before I try to call them from the Flex app. It should be easier to debug that way. So I create the object and use SerializeJSON to make the string to pass to the ColdFusion service method. But I get errors saying such and such property is not found when the service method tries to rebuild the object.

Here’s a simple example that demonstrates the issue

//suppose you have a Java class like so
public class BoundingBox extends DataTransferObject
	implements Serializable
{
	static final long serialVersionUID = -4519523511904622352L;
	//Properties
	private LatLngOnly upperLeftLatLng;
	private LatLngOnly lowerRightLatLng;

	//methods
	public BoundingBox() {}

	public BoundingBox(LatLngOnly upperLeft, LatLngOnly lowerRight)
	{
		this.upperLeftLatLng = upperLeft;
		this.lowerRightLatLng = lowerRight;
	}	

	…
	//various methods including getters and setters
	…
}


<cfscript>
bnd = CreateObject("java", "com.lampo.mapping.vo.BoundingBox").init();
bnd.upperLeftLatLng =
       CreateObject("java", "com.lampo.mapping.vo.LatLngOnly").init(1.23,3.45);
bnd.lowerRightLatLng =
       CreateObject("java", "com.lampo.mapping.vo.LatLngOnly").init(5.67,7.89);
</cfscript>

<cfset str = SerializeJSON(bnd) />
<cfset objBnd =
       CreateObject("java","com.lampo.mapping.vo.BoundingBox").makeFromJSON(str) />

Annoyingly it turns out SerializeJSON is changing some of the property names so they no longer match. While it is true ColdFusion is not case-sensitive, Java is and so capitalizing the first character of the property names causes havoc. If the property is a reference to an object, errors are thrown. If the property is a primitive, the values are not set correctly.

The only solution I have is to alter the JSON string and hard code it as an argument for the method I’m testing. You can do this manually or with some regular expression magic.

<cfset str = SerializeJSON(bnd) />

<cfset pos = REFind('\"[A-Z]',str) />
<cfloop condition="pos NEQ 0">
     <cfset str = Replace(str,Mid(str,pos,2),'"#LCase(Mid(str,pos+1,1))#','all') />
     <cfset pos = REFind('\"[A-Z]',str) />
</cfloop>

<cfset objBnd =
       CreateObject("java","com.lampo.mapping.vo.BoundingBox").makeFromJSON(str) />

Look under the turtle

Developers understand that code reuse necessitates a simple-to-learn interface.  Typically, this leaves us adding another layer of abstraction (aka. “turtle”) to the existing turtle-pile.  This is all wonderful and leaves us feeling warm and fuzzy, except that abstractions are leaky at best.  Usually these new clever interfaces simplify doing whatever you need to do (as long as it is what the API author says you want to do). This means that in order to consume another’s code, I must first learn his or her clever new interface and the interfaces below it.

Recent projects have brought me to the realization that sometimes it’s better to remove a layer of abstraction.  I recently built a solution that involved thermal transfer printing bar codes on pre-printed ticket stock.

The printer uses a proprietary Postscript-like language for creating fields and updating their values throughout the course of a print job.  The printer also came with several half-baked utilities to almost simplify the job of getting it to print what I want, but not quite.  After perusing the developer’s guide, it turned out to be as simple as sending control characters to a parallel port.  I wrote a ‘driver’ in ColdFusion that reads the ticket records from the database and generates a .prn file (just ASCII control codes).  Now I can simply cat the generated file to the fancy printer from any machine that has a parallel port regardless of operating system and without having to install printer drivers.

Albert Einstein is quoted as saying “Things should be made as simple as possible, but no simpler.”  In the example above, I had to learn something new either way—the formatting codes or the libraries that came with the printer.  The libraries failed to simplify the interaction because they did not provide a complete abstraction; I would still have to delve a layer deeper for some of the things I needed.

Think about the abstractions you provide.  Do they really simplify? Will a user of your API need to delve a layer deeper?  If so, you’ve only created an obstacle.

Let’s be really clever developers; let’s learn the abstractions we already have and create clever documentation for other developers.

Adobe Max 2008

We’re headed to Adobe Max, November 17-19!  This year, we’ll have a team of 5 out in California to check out the latest from Adobe.  If you’re planning on attending and would like to meet up, we’ll be wearing “code hope” t-shirts on Monday to make it easy to find us.  Or you can drop a comment here and we can schedule a time for coffee/food.

Also be sure to check out one of our latest projects at the MapQuest booth.  We’ve been having fun with their API and Flex.

BFusion + BFlex ‘08

This September 6-7 a group of us attended the BFusion + BFlex ‘08 conference in Bloomington, IN.  This was a free ColdFusion and Flex hands-on conference sponsored by Adobe, Indiana University, other business and local user groups.

Day One – BFusion

For BFusion, I participated in the Advanced ColdFusion track.  There were some great sessions, but for the purposes of this post, I’ll just recap on two.  Specifically those that were emceed by Elliott Sprehn and Mike Brunt.

Elliott’s presentation on “Frameworks for SOA Platforms” walked us through a custom implementation, his “Shared Services Architecture.” The platform technology stack includes ColdSpring, Model-Glue and Fusebox, and your ORM of choice. Model-Glue is a popular ColdFusion framework that uses a MVC design pattern. Elliot is running a customized version of Model-Glue and basically using the Controller as an ESB of sorts.  For ColdFusion shops who want to migrate to an SOA platform, check out Elliot’s presentation.

Mike Brunt’s session on “ColdFusion in the Enterprise Space – Tuning and Clustering” was perfect timing for us.  We’ve just purchased new web systems hardware and are in the throws of configuring it all and rolling it out.

Day Two – BFlex

This was my first foray into Flex development.  The first day really exceeded my expectations for a free conference and day two followed suit nicely.  The session on event bubbling was really sweet – Flex has done this very nicely.

The other presentation that interesting was “Extending Components.”  I wrote my first Flash app with Remoting and some of the earlier components back in 2002.  We all remember the Pet Store app that the Flash development team at Adobe (then Macromedia) put together.  It’s cool to see how far they’ve come with the entire inheritance structure and extendability in Flex.  Really slick.  Silverlight has it’s work cut out.

The Skinny

To sum it up, it was completely worth going to this conference.  Not every session was not groundbreaking or applicable to our situation, but the ones that were valuable give me the confidence to recommend next year’s event to others (assuming there is one).

Hats off to all of the speakers and volunteers WHO PAID THEIR OWN WAY to be there and put this on for us. It was amazing to know that they did that for those attending and really enjoyed being there.

At one point I was looking for a vending machine to buy bottled water, and couldn’t find one so one of the guys from IU went to his office to get a bottle for me.  Now that’s serving with excellence!

Thanks for sponsoring and hosting such a high-caliber event and making it FREE for those attending!

…oh and did I mention the LOADS of free books/products and free lunch?

ColdFusion Testing Woes With Java Objects

My team is currently working on a Flex-based mapping application. I’m working on the backend using ColdFusion to grab the data from our SQL server, package it in a Java based data transfer object (DTO), and send it to the requesting Flex app. My teammate wrote a great blog entry about the tech stack entitled “Flex, ColdFusion, Java, and BlazeDS: with JSON?“.

One minor glitch showed up while trying to test the ColdFusion method calls. I wrote a simple ColdFusion template to call the methods in the ColdFusion service component. All I wanted to do was simply check the methods were working before I tried calling them from the Flex application. I called the function and used the <cfdump> tag to see what was in the object:

...
<cfset obj = CreateObject("component",
      "components.lampo.mapping.service.MapToolWebServiceImpl") />
<cfset myobj = obj.getPostalCodePolygon(objUser, "37128") />
<cfdump var="#myobj#">
...

Sadly all the <cfdump> tag showed was the names of the methods and properties of the object. I.e. it did not show the property values!

I did a little bit of research and was unable to locate an elegant solution. One possibility would be to simply add a method to each DTO that dumps the current state of the object. But we have over a dozen DTOs and I didn’t really want to pollute them with a debugging method. So I wrote a simple function to display the property values of any object:

<cffunction name="dumpJObj" output="false" returntype="struct" access="public">
       <cfargument name="obj" type="any" required="yes" />
       <cfargument name="depth" type="numeric" required="no" default="1" />

       <cfset var stcRet = StructNew() />
       <cfset var ary = StructKeyArray(obj) />
       <cfloop index="i" from="1" to="#ArrayLen(ary)#">
              <cfif IsDefined("obj.#ary[i]#")>
                     <cfset value = Evaluate("obj.#ary[i]#") />
                     <cfif IsObject(value) AND depth GT 0>
                           <cfset stcRet[ary[i]] = dumpJObj(value, depth - 1) />
                     <cfelseif IsObject(value)>
                           <cfset stcRet[ary[i]] = "depth limit reached" />
                     <cfelse>
                           <cfset stcRet[ary[i]] = Evaluate("obj.#ary[i]#") />
                     </cfif>
              <cfelse>
                     <cfset stcRet[ary[i]] = "" />
             </cfif>
       </cfloop>
      <cfreturn stcRet />
</cffunction>

I wrote the function specifically for the Java based DTOs we are passing. My first thought was to write it using Java reflection but I was able to write it completely in ColdFusion. So I guess it should work for ColdFusion objects as well.

My first version did not handle nested objects so in the next version I added in some recursion. This opened the possibility of an infinite loop if the class has itself as a nested object. I simply added a depth count to prevent this from happening. The depth defaults to 1 so the function will display the values of the top level object and the values of the objects it contains.

To make the function available where I needed it, I made it a part of the service class.

...
<cfset obj = CreateObject("component",
      "components.lampo.mapping.service.MapToolWebServiceImpl") />
<cfset myobj = obj.getPostalCodePolygon(objUser, "37128") />
<cfdump var="#dumpJObj(myobj)#">
...

Flex, ColdFusion, Java, and BlazeDS: with JSON?

We are building a sizable new Flex-based mapping application to support our internal Endorsed Local Provider program. We are passing a significant amount of data between Flex and ColdFusion. So, we architected a service layer that exposes the interface between Flex and CF, and are using BlazeDS to allow us to link Java-based data transfer objects (DTO) on the server side to equivalent Actionscript objects in Flex.

We chose to use Java DTOs rather than CFCs because we found that the Java versions were 15-20 times faster on ColdFusion server. For example, a loop through 1,000 records takes an average of 1.5 seconds:

<cfloop query="rstAddresses">
	<cfset currAddress = CreateObject("component",
                                "components.lampo.mapping.vo.Address") />
	<cfset currAddress.setId(rstAddresses.address_id) />
	<cfset currAddress.setStreet1(rstAddresses.address_line_1) />
	<cfset currAddress.setStreet2(rstAddresses.address_line_2) />
	<cfset currAddress.setCity(rstAddresses.city) />
	<cfset currAddress.setStateCode(rstAddresses.state) />
	<cfset currAddress.setPostalCode(rstAddresses.zipcode) />
	<cfset stcAddresses[rstAddresses.address_id] = currAddress />
</cfloop>

Changing the first line to use a Java object reduces the time to an average of 75 milliseconds:

<cfloop query="rstAddresses">
	<cfset currAddress =
                CreateObject("java", "com.lampo.mapping.vo.Address").init() />
    ...
</cfloop>

I created a custom Java class that generates the Actionscript DTOs from the Java DTOs. We had no trouble sending Java objects to Flex with ColdFusion services through BlazeDS where the parameters passed from Flex were built-in objects, like Strings, Numbers, or Maps. However, when calling a method from Flex that passed one of our custom DTOs as a parameter, the server reported the following error: “Unable to invoke CFC – Could not find the ColdFusion Component or Interface”.

It turns out that if you use ColdFusion services through BlazeDS, meaning that you are using the “cf-object” adapter in your remoting-config.xml, BlazeDS only links ActionScript DTOs with ColdFusion CFC objects on the server. You have to use the “java-object” adapter for it to link Actionscript DTOs to their Java counterparts, but then you have to build your entire service layer in Java, instead of ColdFusion.

Here’s an example of the kind of objects we’re talking about. We have a service method called getPolygons(BoundingBox bb). Flex passes a BoundingBox object, which contains two LatLng objects, and the server returns all of the zip code polygons in that box.

The Java BoundingBox DTO looks like this:

package com.lampo.mapping.vo;
import java.io.Serializable;

public class BoundingBox implements Serializable
{
	//Properties
	private LatLngOnly upperLeftLatLng;
	private LatLngOnly lowerRightLatLng;

	public BoundingBox() {;}

	//Methods
	public LatLngOnly getUpperLeftLatLng() {
		return upperLeftLatLng;
	}
	public void setUpperLeftLatLng(LatLngOnly upperLeftLatLng) {
		this.upperLeftLatLng = upperLeftLatLng;
	}
	public LatLngOnly getLowerRightLatLng() {
		return lowerRightLatLng;
	}
	public void setLowerRightLatLng(LatLngOnly lowerRightLatLng) {
		this.lowerRightLatLng = lowerRightLatLng;
	}

}

And the auto-generated Actionscript version is:

package com.lampo.mapping.vo
{
    import com.lampo.mapping.vo.LatLngOnly;

    [RemoteClass(alias="com.lampo.mapping.vo.BoundingBox")]
    public class BoundingBox
    {
        public var lowerRightLatLng:LatLngOnly;
        public var upperLeftLatLng:LatLngOnly;

        public function getLowerRightLatLng():LatLngOnly
        {
            return this.lowerRightLatLng;
        }
        public function setLowerRightLatLng(in_lowerRightLatLng:LatLngOnly):void
        {
            this.lowerRightLatLng = in_lowerRightLatLng;
        }
        public function getUpperLeftLatLng():LatLngOnly
        {
            return this.upperLeftLatLng;
        }
        public function setUpperLeftLatLng(in_upperLeftLatLng:LatLngOnly):void
        {
            this.upperLeftLatLng = in_upperLeftLatLng;
        }
    }
}

The key to BlazeDS’s ability to link the Actionscript to the Java object is the [RemoteClass(alias="...")] block. From ColdFusion to Flex, a BoundingBox or any other custom object is automatically serialized by BlazeDS and all is well. The problem is going from Flex back to CF with Java DTOs.

To address this, we tried doing some enhancements to the built-in deserialization code, wrapping a ColdFusion DTO around a Java DTO, and decompiling coldfusion.flash.messaging.ColdFusionAdapter to figure out how to make it load Java DTOs. After weighing these ideas we decided to try serializing our Actionscript DTOs into JSON before sending them to ColdFusion. This turned out to be a relatively pain-free solution that maintained our desire to use matching objects on the client and server.

Details

First, we found Adobe’s as3corelib, which includes very handy methods for serializing any object into JSON. Once you download their libraries and include them in your Flex project, it’s as simple as:

import com.adobe.serialization.json.JSON;
JSON.encode(yourObj);

To decode them, we found JSON-lib on SourceForge. Once you have it installed in ColdFusion (see below) you can add a function like this in your Java DTO:

public static BoundingBox makeFromJSON(String inJSON)
	throws Exception
{
	// Place property name/classes in this
        // map to give JSONFunction hints about
	// how to deserialize the JSON
	Map<String, Class> hintsMap = new HashMap<String, Class>();
	hintsMap.put("upperLeftLatLng", LatLngOnly.class);
	hintsMap.put("lowerRightLatLng", LatLngOnly.class);
	BoundingBox bb = (BoundingBox)JSONObject.toBean(
				JSONObject.fromObject(inJSON), BoundingBox.class, hintsMap);
	return bb;
}

Since the BoundingBox object includes properties that are also custom objects, you have to provide a Map that links the property names to the classes, so that JSON-lib can generate the object from the input JSON. This is very handy, and can be called in ColdFusion like this:

<cfscript>
	bbStatic = CreateObject("java", "com.lampo.mapping.vo.BoundingBox");
	jsonBB = "{""upperLeftLatLng"":{""longitude"":-84.5,""latitude"":36}," &
             """lowerRightLatLng"":{""longitude"":-86.79118,""latitude"":39.107}}}";
	realBB = bbStatic.makeFromJSON(jsonBB);
</cfscript>

The JSON-lib setup is a little tricky. Here’s how I installed it into ColdFusion 8:

1. Download JSON-lib and copy the .jar to WEB-INF/lib. The .jar I used is: json-lib-2.2.2-jdk15.jar.
2. Download EZ-Morph and copy the .jar to WEB-INF/lib. The .jar I used is: ezmorph-1.0.5.jar.
3. Download Apache’s Commons-Lang library and copy the .jar to WEB-INF/lib. The .jar I used is: commons-lang-2.4.jar.
4. Download Apache’s Commons-Collections library and copy the .jar to WEB-INF/lib. The .jar I used is: commons-collections-3.2.1.jar.
5. Copy the following JARs from WEB-INF/cfusion/lib to WEB-INF/lib: commons-logging.1.0.4.jar, commons-logging-api.1.0.4.jar, log4j-1.2.12.jar.

With this solution, we’re seeing excellent performance and are maintaining the goal of passing Strongly-typed DTOs between Flex & ColdFusion through BlazeDS using a well-defined service interface.