Independence Delay

W3C announced today the end of the XHTML 2 working group charter.  It has long been apparent that HTML 5 would become a reality, much to my chagrin, but I had hoped deep down that XHTML2 would not die.  You see, HTML 5 is a great tactical move to progress HTML making multimedia easier (among other things) for content authors. However, strategically, I feel this sets back the Web, particularly the semantic Web. I am disappointed, not because XHTML 2 has better features than HTML 5, but because the promotion of HTML 5 over XHTML 2 moves the Web in the wrong direction.

What’s so great about XHTML2

What’s so great about XHTML2? There are some nice cleanups, like replacing <h1-6> with a simple <h>.  The <hr> is replaced with a more semantically-named <separator>. The generalization of the href and src attributes were also pretty exciting. However, the biggest thing is decentralized extensibility. In fact, the W3C mentioned this in their FAQ regarding the end of XHTML2. The very existence of HTML 5 exemplifies why this is a big deal. Decentralized extensibility means we can create our own vocabularies independent of browser implementations.

The current state of things

As a developer, you’re stuck with the elements a user-agent implements.  With HTML, you are limited to what a consortium managed to agree upon and browser authors implemented (somewhat) consistently. The needs of content authors and the capabilities browsers have outgrown the existing centralized vocabularies of HTML 4 and XHTML 1.  HTML 5 addresses this problem with a brute force method: keep improving the centralized vocabulary, and try to keep up with the developer community.

Where we could be headed

If we currently live in the era of the Web browser, we ought to be headed toward the era of the XML browser.With XML, we have more liberty. We can create XML documents with our own vocabularies. We can even define the vocabularies through DTDs or other schemata.  We can mix and match our vocabularies with others. We can define the layout and presentation of new languages with CSS and XSL-FO. Better still, we can define what our new elements and attributes mean through RDF and its related technologies. This is the key.

Every time I mention this, I get the same question, “How would a browser know that my attribute mynamespace:foo@fetcherator is the same as html:a@href?” This is where RDF comes in. With semantic technologies, we can define our vocabularies in such a way that a browser will know.

Why can’t we head there with HTML 5?

We can. In fact, I believe we will (though perhaps more slowly now). My concern is the motivation behind the HTML enhancements in HTML 5 vs XHTML 2. XHTML 2 was a continuation of the W3C’s efforts to bring the benefits of XML into HTML, while HTML 5 seems to bring the benefits of new user agent capabilities into HTML. We need to decouple the user agent capabilities from our document vocabularies. This is true independence.

We can still do this.  The future may still be bright. I hope that we do not lose the vision as we postpone liberty for the comforts afforded by the latest iteration of a centralized HTML.

Adobe ColdFusion Rocks Lampo

On June 18th we were honored to have had the 2009 Adobe ColdFusion User Group Tour on our very own stomping grounds in Brentwood, TN.  The event was held in our conference center and we had a great turn out.  Greg Wilson, an Adobe Product Evangelist, demo’ed many new features in the upcoming release of ColdFusion 9.  The majority of our crew attended and took a lot away from it.  Here are some of their thoughts and photos with highlights from the exciting night!

“I’m particularly pleased with the ability to do much more in <cfscript /> that ever before. As a Java developer, it makes ColdFusion much more appealing. Also, Hibernate integration is particularly interesting – that should improve productivity for data-driven apps dramatically.”

Doug Smith – Sr. Web Programmer

“It was great to see how much Adobe has their finger on the pulse of the development community.  Many of the new features for CF9 are things that just in the past week I had heard someone say, ‘Wouldn’t it be great if you could do that in Coldfusion?’ and sure enough, soon you’ll be able to.  Getting that behind-the-scenes perspective from Greg was really cool.”

Ty DeLong – Web Programmer

“The most interesting parts of the event for me were seeing the demonstration of the new, long-awaited Bolt IDE.  It has a lot of nice features that should improve development.  I also enjoyed the new features of the next version of the ColdFusion language. The presenter explained that Adobe has added a lot of little features to make syntax more consistent throughout ColdFusion. They’ve also added some interesting controls, ie. the DataGrid, which might help us to do some standard things more quickly.”

Ron Coffman – Marketing Web Programmer

“The event was a great example of Adobe stepping up once more and retooling ColdFusion into an industry-leading Rapid Application Development language. It was awesome to hear first-hand about some features that I’ve wanted to see for quite a while!”

Tim Kucejko – Web Programmer

“Several great features were presented. Bolt looks like a nice upgrade to CFEclipse. I like the new tags that simplify using JavaScript interface components. I also like the enhancements being made to the cfscript tag. But best of all: ternary operator …”

Jon Fouss – Sr. Web Programmer

Ugly Shirt Day 2009

Our team has been working really hard lately on redesigning the Dave Ramsey site (which is going to look amazing!  keep an eye out!!!).  The other day, a few of us were discussing the new site when one of the guys complimented another on his choice of attire for the day.  The recipient of the adulation was our very own Jon Wolski.  He is the embodiment of retro chic and that particular day was sporting a fabulous Hawaiian button-up.

In a half-joking, half-serious manner it was suggested that we sponsor a Hawaiian shirt day in honor of those who are “encouraged” to do so regularly by corporate america to support “team spirit.”  The idea bloomed to wearing some silly shirt every Friday for the month of June.

Well, last Friday we had Ugly Shirt Day 2009!  Don’t we look awesome?  So with all this shirt ugliness going around we decided to vote for the worst.

Here are the results!

That cool dude posing in the front, Jim, WON with his “Dream” shirt and Kelly, off to the far right received Runner-Up with his amazing “Three Wolves” shirt.  For the prize Jim received an official “Ug Mug” and ugly bird figurine trophy!  Kelly won an ugly hat.  Congratulations boys for making ugly look easy!  I didn’t do too bad myself, ehhh?!

Cheers to all,

Katie

“An Untamed Spirit” (It’s on my classy Sturgis 2001 shirt)

Have a great week everybody!

P.S. Chris took these radical photos!  Way to go dude!

You know... just being ourselves!

2009 Adobe ColdFusion User Group Tour

In just a few short weeks, we’ll have the privilege of hosting Adobe and the Nashville Cold Fusion User Group here at our facilities in Brentwood, TN for the 2009 Adobe ColdFusion User Group Tour.

Here’s the official post from the NCFUG website:

We are SUPER excited to have the opportunity to get the surrounding ColdFusion community together to talk about the next release of ColdFusion. This year our presenter will be Greg Wilson, an AIR, ColdFusion, Flex, and LiveCycle ES evangelist at Adobe.

Greg will be coming to Nashville to talk about the new features in ColdFusion, how ColdFusion integrates with other technologies like AIR, Flex, and LiveCycle, and he’ll be demoing many of the new features in the next version.

We’ll have food, drink, and some awesome prizes/giveaways, so you do not want to miss this special event. Finally, I’m happy to report the great folks at Dave Ramsey / The Lampo Group, Inc. will be hosting this event at the Russ Carroll Financial Peace Conference Center just minutes from downtown Nashville. They were a big reason this same event in 2007 was a success and we are excited they’ll be part of it again.

As we work with Adobe to nail down some final details we will update this page. So go ahead and RSVP and check back for additional information.

WHEN:
Thursday, June 18, 2009 at 7:00pm CDT

WHERE:
Russ Carroll Financial Peace Conference Center (map link below)

SPEAKER:
Greg Wilson – Adobe evangelist for AIR, ColdFusion, Flex and LiveCycle ES

RSVP Info:
In order to help us plan the event we kindly ask you to RSVP by pressing the link below. It only takes a few seconds!

Location Info
1749 Mallory Lane, Brentwood, Tennessee 37027

Map of Location

RSVP for this Meeting

We hope you can come out and join us!

Custom Fonts in ColdFusion

We have a couple non-standard fonts that we use for some of our our FPU class materials. For our new Coordinator Resource Center we wanted to use these same fonts in a server generated PDF file. However, this was the first time we’ve actually used a non-standard font with ColdFusion. This turned out to be a much more difficult task than it would seem. During this process I ran into a few issues. (These apply to ColdFusion MX 7.02)

Font Family vs Font Face
Within the True Type Font file, there are definitions for the “Font Family” and the “Font Face.” For some reason, ColdFusion would not read the file unless these were set to the exact same string. I used a font editor to modify these. In addition, if the font file contains different a “Windows Name” and “Mac Name” you might want to make those the same as well.

System vs. User Defined File Location
There is a difference in where you place the files. If you happen to be running ColdFusion on a Windows box, you’re better off just installing the fonts as you normally would through the control panel. Developing locally, I didn’t have any problems with that. However, our server runs Linux and I needed to set a custom path to tell ColdFusion where to look for the fonts. This is easily done through the Administrator control panel.

However, unless you have this hot fix installed, it will never read the fonts and never give you an error. The admin tool will show you they are installed, but when you try to generate a PDF using them, it will always fall back to a system font. The documentation on this hot fix gives you no indication that it will fix this problem, and it is not included in any other service pack.  However, this solved the problem for us.

Font Types
Supposedly, ColdFusion will support the following font types:
TTF (True Type Font)
TTC (True Type Collection)
OTF (Open Type Font)
AFM (Adobe Font Metrics)
However, I ran into an OTF font where (no matter what I did) I could not get the PDF to output that font. It would keep falling back to the system font. I ended up using [FontForge] to convert it from OTF to TTF.

Plan your escape

URL encoding is not character/entity encoding.

This should go without saying, but I frequently see this confused by experienced developers, especially when working with dynamic/loosely typed languages.

URL encoding is for URLs (URIs to be more generic).  The only time to URL-encode a string is when it is part of a URL.  JavaScript provides encodeURI(), encodeURIComponent(), decodeURI, and decodeURIComponent().  In ColdFusion, you can use the  URLEncodedFormat() and URLDecode() functions.  PHP provides urlencode(), rawurlencode() and their decode counterparts.

Entity encoding is used for representing characters in a document that lie outside of the document character set or have a special meaning within the document.  For example In XML, &, <,>,”,and ‘ have to be encoded as entities (’&, >, etc.) in the document source code.  Typically, characters outside of “low” ASCII need to be encoded as well.  In client-side code, you typically need not worry about entity-encoding. You’re working with a DOM, not document source code, so entity expansion/subsitituion has already been done.

ColdFusion is a little tricky on this one.  There is no equivalent to PHP’s htmlentities.  You basically have two options, HTMLEditFormat and XMLFormat.  The former will encode characters with special meanings, but it misses high-ASCII (and higher).  The latter will encode high-ASCII, but will not use special HTML entity names.  It’s for (the more generic) XML after all.  XMLFormat escapes characters using character entity references.

The concept applies outside of this concrete example, but this is the example that led me to channel my angst into what I hope is a helpful guide post for others.  In fact, you’ll notice that two paragraphs above, Word Press has transformed my double and single quotes to right-double and left-single quotes, respectively.  What are some other escaping/transforming pitfalls you’ve seen?

User-Centric Development

The keynote at DevLearn 2008 this past past fall was given by Dan Roam, the author of The Back of the Napkin. Dan spoke about the importance of thinking visually, especially when problem solving.

His presentation spurred a conversation between myself and the other Lampo attendees at the conference: Jon Shearer and Michael Finney. We were discussing how bad things can happen when software engineers find a neat “feature” they can work into an app, regardless of whether the users want or need it.

This brought Finney to a perfect, real-life example of this concept in action. He had recently purchased a new Gateway laptop with a built-in 802.11n wireless network adapter. When using the wireless, however, he had noticed that it would occasionally slow to a crawl. It’s never terribly convenient to troubleshoot this type of problem, especially when so many components are involved (ISP, broadband modem, wireless router, laptop hardware, OS…) and the problem is intermittent.

Well, last weekend, he had a guest in his house with a laptop, and he was able to do a side-by-side speed test during one of these episodes, and he discovered that while the other laptop was getting about 20mbps down, his was getting about 20kbps down.

This took the network out of the troubleshooting stack, so he really dug into the laptop to locate the source of the problem. Before long, he found a “feature” that is new to Windows Vista.

As with prior versions of Windows, you can create and customize power schemes to use when on battery power in order to preserve battery life by throttling the power that certain components use. It is common to scale back the processor speed or the LCD brightness to achieve longer batter life.

Well, Vista has added Wireless Network Adapter Throttling into the default power saver scheme, effectively crippling your network speeds in the name of battery preservation. Let’s think this through for a minute. I want to preserve battery life, so having to sit and wait 1000x longer for a website to come up (all while my LCD and processor burn through battery while I sit and do nothing) is the answer?

Just because you can do something as a developer with the technology you have, doesn’t mean you should.

And just in case you got to this post by searching for an answer to your network slowdown woes, here’s how you change this setting in Vista. Control Panel -> Hardware & Sound -> Power Settings -> Change Plan Settings (on your currently selected power plan) -> Change advanced power settings -> Wireless Adapter Settings -> Power Saving Mode. That’s right, 7 clicks deep, one of which includes clicking on “Sound”. *sigh*

The Sweet Java Topology Suite – Part II

In a previous post, we described how we started using the Java Topology Suite (JTS) to manipulate postal/zip code polygons that we are viewing in an application built on MapQuest’s Flex API. Since then, we have added the ability to join multiple postal codes into territories. Sometimes over 1,000 postal code polygons will be combined to form a single territory.

We ran into two significant technical hurdles. First, MapQuest’s API doesn’t support polygons with inner holes. So, a donut-shaped polygon would just look like a circle, with no hole in the middle. The other problem was that some of the postal codes were so complicated that the unify process would fail.

Union of postal code polygons with a hole in the middle

Union of postal code polygons, missing the hole in the middle

If you read the other article, you saw that we did use JTS to simplify polygons (by reducing the number of points that make up the polygon). However, we didn’t end up using those in production because the edges of the simplified polygons would not line up. They end up looking like broken glass, because the simplify process had no regard for adjacent polygon edges.

Simplified polygons with edges that don't line up

So, we set out on an adventure to simplify the polygons so that the edges of the simplified postal codes matched up. We received some very responsive and helpful guidance from Martin Davis, one of the principle developers of JTS. He also pointed us to the open source tool OpenJUMP, which he also helped to build. Source code from that tool was very helpful as we created our own automated simplification process.

Here’s the simplification process in a nutshell:

1. Convert the MapQuest postal code polygon data for the current patch (like the lower 48 states) to Well-Known Text (WKT) and save each postal code polygon to an individual file on the file system. For the lower 48 states, this resulted in more than 41,000 files. Here is an unsimplified version of the few polygons we’ll simplify in this example:
   

   Original, unsimplified postal code polygons

2. Read all of the WKT files, one per postal code, and store them as JTS Geometry objects in a collection. To support step six (below), we store the postal code in the geometry object using the very handy Geometry.userData property. That way, each original/source geometry remembers what postal code it represents.
3. Use JTS to convert the polygons to merged LineString objects. This creates a collection of the outlines of every polygon, where the common polygon edges become a single line.
   

   Extracted border lines of original polygons

4. Use JTS to simplify the merged LineStrings by reducing the number of coordinates that define each line. Our code iterates across every merged LineString and uses JTS’s DouglasPeuckerSimplifier with a simplify tolerance of 0.01.
   

   Simplified polygon border lines

5. Use JTS to create polygons from the simplified LineStrings. The primary JTS class was the magic Polygonizer class, along with code from OpenJUMP that prepared the line data for the Polygonizer.
   

   New polygons made from simplified lines

6. Now the tough part. We have a collection of simplified polygons, but they aren’t linked to any postal codes, so we can’t find the polygon and use it in our application. We needed to match the simplified polygon with the original. Since this is among the most involved processes, I’ll describe it in a bit more detail:
   1. Add each of the original polygons to a JTS SpatialIndex called STRtree. The STRtree provides a quick query interface to find polygons that fall within a spatial constraint.
   2. Iterate through each of the simplified polygons, and:
      1. Query the STRtree to find all of the original polygons that touch the envelope (bounding rectangle) of the current simplified polygon.
      2. Find the polygon in that set which has the smallest distance between its center point and the simplified polygon’s center point.
      3. Once the best matching simplified polygon is found, we copy the postal code from the original Geometry’s userData.
      4. Some simplified polygons have no match in the original set because of holes, so those non-matches are thrown out in this process.
   3. Now that each simplified polygon has been identified as matching a postal code, we write new WKT files for each postal code. Our code that writes these files automatically creates MultiPolygon objects for those postal codes that are made up of more than one polygon.
   

   Simple polygons that remain after match with originals

In order to run this process on the lower 48 United States, I had to allocate 7GB of my 8GB of RAM to the JVM so that all 41,000 polygons could be simplified at the same time. Fortunately, it’s worth the time to build. Here are the number of coordinates needed to represent all of the polygons for the three areas, both originally and after simplification, along with the savings realized:

	Coordinate Count
	Original	Simplified	Reduction
Lower 48 United States	6,276,000	544,000	12x smaller
Alaska	262,000	15,000	17x smaller
Hawaii	72,000	960	75x smaller

Here’s a larger area of polygons, before and after simplification:

Original postal code polygon sample

Simplified postal code polygon sample

In order to create polygons that maintain any holes in the middle with MapQuest’s polygon API, we used JTS to cut a small slice between any inner features and the exterior of the polygon. This leaves a line in the middle of the polygon, but it’s more acceptable than no hole at all. Hopefully MapQuest will support polygons with inner holes in a later release. In fact, it would be really cool if MapQuest would incorporate other structures and features from JTS, including native WKT support.

Simplified postal codes on map

Territory on map with hole enabled by slice

We are very grateful for the Java Topology Suite and the polygon processing it allowed us to complete. The project we’re building for Dave’s Endorsed Local Provider program will be much more successful with these improvements.

How Much Memory Does This Take?

Ever wonder how much memory an object takes?

The web has lots of  great theoretical answers to that question.

Theories are great, and they give lots of great insights into how a Java compiler works. But what if things aren’t perfectly clean-cut? There are plenty of reasons to want proof including the following possibilities:

• Say you use the Spring Framework or some other mechanism where your objects aren’t composed until runtime
• Maybe you use a library/jar/package/swf/swc/etc, you don’t have access to the underlying code, and you’re trying to weigh its memory footprint
• Maybe your objects are composed differently in different states
• Maybe you use a Rapid Application Development language such as  ColdFusion or PHP and want to see how much overhead they bring along
• Maybe you’re just a hard-core detailed person who wants to prove that the theory really matches reality

We’ve had to troubleshoot ColdFusion apps for memory usage, and we’ve found this approach to be helpful:

1. Call for garbage collection (if applicable)
2. Take a snapshot of how much memory is used
3. Instantiate a whole bunch of copies of the class in question, and stuff them into an array
4. Call for another garbage collection (if applicable)
5. Take another snapshot of how much memory is used
6. Subtract the first memory reading from the second, divide by the number of objects created, and you have a pretty good approximation of how much memory each object takes

As a side-note, the more objects you create in your test, the more your test will drown out background noise from other activity, and the more accurate your reading will be on each object.

After researching this method, we honed it a bit and made some useful discoveries. For example, in ColdFusion 7, each CFC instantiated takes up ~2kB. On top of that, each <cffunction></cffunction> in that cfc takes up an additional ~150 bytes.  So the abc.cfc “class” (found below) with 3 empty methods takes up ~2.3kB. (disclaimer: I’ve heard but not yet personally verified that memory usage improves in CF8 and beyond.)

In our scenario, we had CFCs inheriting a data persistence layer including about 50 methods, our CFCs included about 50 more in addition to member data in the variables scope. Without fully understanding the under-the-hood functionality of ColdFusion 7, we were instantiating 30 CFCs per user at 20kB each (~half a meg) for a few thousand users. Then we wondered why we kept eating through our available memory so quickly.

After making this discovery we spent a few days re-factoring some code hoping to use resources better. We limited the inheritance of the persistence layer to where it was necessary. We re-factored the code to use ColdFusion queries rather than CFCs (queries wound up being MUCH lighter than similar CFCs). When we launched the changes our application used about 40% less memory.

As hinted earlier, this algorithm can also be used to analyze memory usage by basic Java components. One question I’ve dabbled with is how much memory java code takes. I may cover that subject in another post someday, but for now I’ll leave that as an exercise for the reader. In the meantime, here’s a ColdFusion script that implements the algorithm described above.  Enjoy!

<!---this part executes in a cfm--->

<cfset runtime = CreateObject("java","java.lang.Runtime").getRuntime() />
<cfset myArray = arrayNew(1) />
<cfset intNumberOfObjectsCreatedPerLoop = 10 />
<cfset intTotalNumberOfObjectsCreated = 0 />
<cfset intX = 0 />
<cfset intBytesBeforeTest = 0 />
<cfset intBytesAfterTest = 0 />
<cfset intBytesCreated = 0 />
<cfset intBytesIn1MB = 1048576 />
<cfset memoryThresholdInMB = 50/>
<cfset obj = "" />

<cfset runtime.gc() />
<cfset intBytesBeforeTest = runtime.totalMemory() - runtime.freeMemory() />
<cfloop condition="intBytesCreated lt (intBytesIn1MB * memoryThresholdInMB)">
<cfset intTotalNumberOfObjectsCreated = intTotalNumberOfObjectsCreated +
		intNumberOfObjectsCreatedPerLoop />
	<cfloop from="1" to="#intNumberOfObjectsCreatedPerLoop#" index="intX">
		<cfset obj = CreateObject("component", "abc") />
		<cfset arrayAppend(myArray, obj) />
	</cfloop>
	<cfset intBytesAfterTest = runtime.totalMemory() - runtime.freeMemory() />
	<cfset intBytesCreated = intBytesAfterTest - intBytesBeforeTest />
</cfloop>
<cfoutput>
<cfset runtime.gc() />
#intBytesCreated / 1024# kilobytes of objects created.<br>
Each CFC takes approximately :
#(intBytesAfterTest - intBytesBeforeTest) / intTotalNumberOfObjectsCreated / 1024#kB
</cfoutput>
<!--- this part resides in a separate file (abc.cfc) in the same directory --->

<cfcomponent name="abc">
	<cffunction name="a" output="false"><!--- Really important code here. ---></cffunction>
	<cffunction name="b" output="false"><!--- Really important code here. ---></cffunction>
	<cffunction name="c" output="false"><!--- Really important code here. ---></cffunction>
</cfcomponent>

The Hits Just Keep On Coming

It’s been an interesting year for us webmonkeys.

In the last several months a few big banks, insurance companies, and/or financial companies were deeper into sub-prime mortgages than they should have been. The US government stepped in to either loan money (AIG) or take over (Fannie Mae, Freddie Mac) said organizations. The aforementioned steps and those that followed have since blown open the door for much more intervention.

If you’re very familiar with Dave Ramsey, you probably know that we don’t like debt very much around here. To try and stop the one of the first multi-billion dollar bailouts from becoming law, Dave went on 32 different radio and tv shows within 36 hours talking about The Common Sense Fix as an alternative to buying hundreds of billions of bad debts.

This all came together very quickly. Needless to say, it left us in the exciting position of being able to serve more people than our website was comfortably equipped to handle at the time.

At that point in time, many of the pages on our website are generated dynamically every time someone visits. We quickly took static “snapshots” of the high-traffic dynamic pages and put them out there to speed load times. In addition, we moved a lot of our images, .css, .js, and other static files out to be served by Akamai instead of by our local servers. It’s a bit more difficult to update things when Akamai hosts them, but they have a ginormous amount of capacity in their network. To use another example from the industry, Yahoo uses Akamai for most of their static content as well. We were pleasantly surprised by how dramatic of a speed increase came from using them. Perhaps even more exciting, Akamai’s bandwidth price is less expensive than our more traditional hosting arrangements.

What else did we do to prepare? In addition to upgrading some hardware, we also looked at our code to see how we could reduce database usage. We wrote this query to analyze the top 50 queries performing IO on our database:

select top 50
qs.total_worker_time / execution_count as avg_worker_time,
substring(st.text, (qs.statement_start_offset/2)+1, ((
case qs.statement_end_offset
when -1 then datalength(st.text)
else qs.statement_end_offset
end – qs.statement_start_offset)/2) + 1) as statement_text,*
FROM sys.dm_exec_query_stats as qs with (nolock)
cross apply sys.dm_exec_sql_text(qs.sql_handle) as st
order BY avg_worker_time desc

As a quick disclaimer, the query above takes a cross section of the I/O reads executed over the last 10 minutes. We did notice significant differences in the pie charts generated at different times of the day. If you’re going to be investing a lot of time or money into making optimizations, you need to run several of these reports at different times and do some experimentation of your own to see what makes the most sense for you.

We took the results of the above query, we grouped the queries by application, and we made a quick excel pie chart like this one:

Database Usage by Application

With this data, we were then able to dig down into the code and analyze which activities were causing the heaviest database load. In a pinch we were able to quickly shut off a scheduled task and optimize another heavy piece of code to reduce our database IO by 50%.

After the first round of optimizations, we ran the report again and created reality-based plans for further optimization should it be needed.

Getting actual concrete data on the queries actually causing I/O reads was empowering. At the outset of this we brainstormed ways to increase capacity. We proposed some fairly involved projects to optimize a few parts of the site that we knew executed queries. Thankfully we waited for real data before acting, because none of the ideas we initially proposed would have fixed any of the 50 slowest queries.

We made it through this event without serious disruption to our web visitors. However, the experience is really making us think twice about what would happen if we were ever to be Dugg or Slashdotted in a big way.

What an INCREDIBLE problem to have!