Treating Customers Well

We send millions of emails every year trying to help people. To help us have unique conversations with all those people simultaneously, we rely on a vendor named Exact Target. Around here we tend to push the envelope, and with very few exceptions they’ve been able to keep pace with us every step of the way.

Not long ago, we had a service interruption. We use their API to do a lot of different things, and several times over the period of a week, our API requests started failing. We quickly found and used a workaround to get by it the first few times, but when it kept happening we felt like we needed a deeper investigation. After some work, they found the issue on their end, they apologized profusely, and corrected it. Two days later, I arrive at my desk and find a package from them. It included a handwritten note, a $5 Starbucks gift card, and orange-flavored candy. (Everything they do is orange.)

I sent our contact an email thanking them for the gift and got this reply:

Tim, I’m glad you enjoyed the token of our consolation for the recent issues! Obviously candy oranges do not negate the product issues, but we hope it proves our appreciation for your understanding and solid partnership. We also appreciate the feedback and kind remarks – it always helps make a day! Please let us know how we can continue to help and support your team in the future. Best, Megan

Our team strives to super-serve people and we like to build relationships with vendors that do the same.  The way Exact Target handled this situation impressed me. It’s awesome to see things done at such a high level of excellence. I’m so impressed with it that I’m showing this care package to people around our office hoping that we can learn from the example. How does your company super-serve your customers?

What a Year

Our existing website was about 5 years old. After 5 years and uncounted utterances of  “Hey! We have a great new product that needs exclusive top billing on our website!” our web presence was… a bit busy.

We took our time, did our homework, researched, discussed, and planned a long project. In March, Right after we started into what became thousands of man hours, we had ANOTHER cool idea. Right in the middle of a season of economic uncertainty Dave led the team to do an event giving reasons to be hopeful. In early March we had a full workload and an unrelated idea. 2 months later on April 23rd, we had a fully functional website serving 1,000,000 people watching streamed video at 6,000 volunteer locations.

Of course that means that we pushed back launching the main site… right? Well,  no.  Did we work a ton of overtime? Well, we did some but it wasn’t 2 months worth. The new website is now up at www.daveramsey.com, and the side idea is up at www.townhallforhope.com.

I love that we did all of it without staying at the office so much that we forgot who our kids are. It’s been a great year. I’m excited to see what comes next!

How Much Memory Does This Take?

Ever wonder how much memory an object takes?

The web has lots of  great theoretical answers to that question.

Theories are great, and they give lots of great insights into how a Java compiler works. But what if things aren’t perfectly clean-cut? There are plenty of reasons to want proof including the following possibilities:

• Say you use the Spring Framework or some other mechanism where your objects aren’t composed until runtime
• Maybe you use a library/jar/package/swf/swc/etc, you don’t have access to the underlying code, and you’re trying to weigh its memory footprint
• Maybe your objects are composed differently in different states
• Maybe you use a Rapid Application Development language such as  ColdFusion or PHP and want to see how much overhead they bring along
• Maybe you’re just a hard-core detailed person who wants to prove that the theory really matches reality

We’ve had to troubleshoot ColdFusion apps for memory usage, and we’ve found this approach to be helpful:

1. Call for garbage collection (if applicable)
2. Take a snapshot of how much memory is used
3. Instantiate a whole bunch of copies of the class in question, and stuff them into an array
4. Call for another garbage collection (if applicable)
5. Take another snapshot of how much memory is used
6. Subtract the first memory reading from the second, divide by the number of objects created, and you have a pretty good approximation of how much memory each object takes

As a side-note, the more objects you create in your test, the more your test will drown out background noise from other activity, and the more accurate your reading will be on each object.

After researching this method, we honed it a bit and made some useful discoveries. For example, in ColdFusion 7, each CFC instantiated takes up ~2kB. On top of that, each <cffunction></cffunction> in that cfc takes up an additional ~150 bytes.  So the abc.cfc “class” (found below) with 3 empty methods takes up ~2.3kB. (disclaimer: I’ve heard but not yet personally verified that memory usage improves in CF8 and beyond.)

In our scenario, we had CFCs inheriting a data persistence layer including about 50 methods, our CFCs included about 50 more in addition to member data in the variables scope. Without fully understanding the under-the-hood functionality of ColdFusion 7, we were instantiating 30 CFCs per user at 20kB each (~half a meg) for a few thousand users. Then we wondered why we kept eating through our available memory so quickly.

After making this discovery we spent a few days re-factoring some code hoping to use resources better. We limited the inheritance of the persistence layer to where it was necessary. We re-factored the code to use ColdFusion queries rather than CFCs (queries wound up being MUCH lighter than similar CFCs). When we launched the changes our application used about 40% less memory.

As hinted earlier, this algorithm can also be used to analyze memory usage by basic Java components. One question I’ve dabbled with is how much memory java code takes. I may cover that subject in another post someday, but for now I’ll leave that as an exercise for the reader. In the meantime, here’s a ColdFusion script that implements the algorithm described above.  Enjoy!

<!---this part executes in a cfm--->

<cfset runtime = CreateObject("java","java.lang.Runtime").getRuntime() />
<cfset myArray = arrayNew(1) />
<cfset intNumberOfObjectsCreatedPerLoop = 10 />
<cfset intTotalNumberOfObjectsCreated = 0 />
<cfset intX = 0 />
<cfset intBytesBeforeTest = 0 />
<cfset intBytesAfterTest = 0 />
<cfset intBytesCreated = 0 />
<cfset intBytesIn1MB = 1048576 />
<cfset memoryThresholdInMB = 50/>
<cfset obj = "" />

<cfset runtime.gc() />
<cfset intBytesBeforeTest = runtime.totalMemory() - runtime.freeMemory() />
<cfloop condition="intBytesCreated lt (intBytesIn1MB * memoryThresholdInMB)">
<cfset intTotalNumberOfObjectsCreated = intTotalNumberOfObjectsCreated +
		intNumberOfObjectsCreatedPerLoop />
	<cfloop from="1" to="#intNumberOfObjectsCreatedPerLoop#" index="intX">
		<cfset obj = CreateObject("component", "abc") />
		<cfset arrayAppend(myArray, obj) />
	</cfloop>
	<cfset intBytesAfterTest = runtime.totalMemory() - runtime.freeMemory() />
	<cfset intBytesCreated = intBytesAfterTest - intBytesBeforeTest />
</cfloop>
<cfoutput>
<cfset runtime.gc() />
#intBytesCreated / 1024# kilobytes of objects created.<br>
Each CFC takes approximately :
#(intBytesAfterTest - intBytesBeforeTest) / intTotalNumberOfObjectsCreated / 1024#kB
</cfoutput>
<!--- this part resides in a separate file (abc.cfc) in the same directory --->

<cfcomponent name="abc">
	<cffunction name="a" output="false"><!--- Really important code here. ---></cffunction>
	<cffunction name="b" output="false"><!--- Really important code here. ---></cffunction>
	<cffunction name="c" output="false"><!--- Really important code here. ---></cffunction>
</cfcomponent>

The Hits Just Keep On Coming

It’s been an interesting year for us webmonkeys.

In the last several months a few big banks, insurance companies, and/or financial companies were deeper into sub-prime mortgages than they should have been. The US government stepped in to either loan money (AIG) or take over (Fannie Mae, Freddie Mac) said organizations. The aforementioned steps and those that followed have since blown open the door for much more intervention.

If you’re very familiar with Dave Ramsey, you probably know that we don’t like debt very much around here. To try and stop the one of the first multi-billion dollar bailouts from becoming law, Dave went on 32 different radio and tv shows within 36 hours talking about The Common Sense Fix as an alternative to buying hundreds of billions of bad debts.

This all came together very quickly. Needless to say, it left us in the exciting position of being able to serve more people than our website was comfortably equipped to handle at the time.

At that point in time, many of the pages on our website are generated dynamically every time someone visits. We quickly took static “snapshots” of the high-traffic dynamic pages and put them out there to speed load times. In addition, we moved a lot of our images, .css, .js, and other static files out to be served by Akamai instead of by our local servers. It’s a bit more difficult to update things when Akamai hosts them, but they have a ginormous amount of capacity in their network. To use another example from the industry, Yahoo uses Akamai for most of their static content as well. We were pleasantly surprised by how dramatic of a speed increase came from using them. Perhaps even more exciting, Akamai’s bandwidth price is less expensive than our more traditional hosting arrangements.

What else did we do to prepare? In addition to upgrading some hardware, we also looked at our code to see how we could reduce database usage. We wrote this query to analyze the top 50 queries performing IO on our database:

select top 50
qs.total_worker_time / execution_count as avg_worker_time,
substring(st.text, (qs.statement_start_offset/2)+1, ((
case qs.statement_end_offset
when -1 then datalength(st.text)
else qs.statement_end_offset
end – qs.statement_start_offset)/2) + 1) as statement_text,*
FROM sys.dm_exec_query_stats as qs with (nolock)
cross apply sys.dm_exec_sql_text(qs.sql_handle) as st
order BY avg_worker_time desc

As a quick disclaimer, the query above takes a cross section of the I/O reads executed over the last 10 minutes. We did notice significant differences in the pie charts generated at different times of the day. If you’re going to be investing a lot of time or money into making optimizations, you need to run several of these reports at different times and do some experimentation of your own to see what makes the most sense for you.

We took the results of the above query, we grouped the queries by application, and we made a quick excel pie chart like this one:

Database Usage by Application

With this data, we were then able to dig down into the code and analyze which activities were causing the heaviest database load. In a pinch we were able to quickly shut off a scheduled task and optimize another heavy piece of code to reduce our database IO by 50%.

After the first round of optimizations, we ran the report again and created reality-based plans for further optimization should it be needed.

Getting actual concrete data on the queries actually causing I/O reads was empowering. At the outset of this we brainstormed ways to increase capacity. We proposed some fairly involved projects to optimize a few parts of the site that we knew executed queries. Thankfully we waited for real data before acting, because none of the ideas we initially proposed would have fixed any of the 50 slowest queries.

We made it through this event without serious disruption to our web visitors. However, the experience is really making us think twice about what would happen if we were ever to be Dugg or Slashdotted in a big way.

What an INCREDIBLE problem to have!

Little Changes and HUGE Effects

iTunes operates similar to a Digg type website. In this case, people vote by subscribing to a podcast inside iTunes. The more people click on your podcast, the higher the ranking goes. The top spot on the podcast ranking can be valuable real estate to hold, but how does one become (and stay) king of the mountain?

Of course, it really helps if you have a popular website that can drive people to your podcast. Then you can use a link like the one below (every podcast has their own unique url) to send people to the iTunes Store.

http://phobos.apple.com/WebObjects/MZStore.woa/wa/viewPodcast?id=77001367

The iTunes Podcast Subscription Page (Red Circle Added)

The above link brings up iTunes and preloads it to a screen like this one. Where iTunes ratings are concerned, it’s all about clicking that last button.

It’s also entirely time-based. You can have 3 Million podcasters sign up over the course of a year. Assume that all of them are avid listeners who hear the show every day. That would be 10,000 people/day. However, say a TV show ends and posts a snippet on iTunes, and they get 150,000 people to download the snippet over 3 days. Even though the theoretical TV show only has 5% of your theoretical audience, they would have 40,000 more subscribers per day and presumably take the #1 slot for 3-4 days.

It’s an interesting animal. A while back, we added images to our podcast feed, and almost immediately we jumped to #1 in the business category, and for the first time we broke the top 100 overall. Late last year, we started to combine 2 feeds, and jumped from an average ranking around 80 to an average ranking around 40 – and we have more changes coming. (I’ll detail what we’ve done in a future post.)

When you start working with groups of people in the millions, small changes can make a HUGE difference. We’ve seen it here many times in various forms, but to me it’s like a jet leaving the runway – it still amazes me every time.

Foam Ball Friday

Imagine this… The end of the workday is almost here. It’s Friday. You’ve had a lot of fun this week working on cutting-edge technologies… But at the same time you’ve been bouncing around between several different projects all week. First off, you had the urgent media event that you need to be ready for, you have half a dozen meetings about an up-and-coming revolutionary project that’s going to turn the way you’ve done things on it’s ear, and of course there’s always the few minor things you do to help your customers or help maintain existing code.

Are They Stress Balls... Or Future Projectiles?

 
Needless to say, you’ve had a great week, but you’re also ready for a weekend off. Then it starts. It begins as a trickle. Slowly it becomes more frequent, and the next thing you know you’re surrounded by flying foam stress balls.
 

Foam Ball Friday has been a web-team tradition for 3 years now. It started off as a promotion. We were giving out foam footballs at events with the slogan “Tackle Debt with MyTotalMoneyMakeover.com.” Then the promotion ended and we were left with the challenge of finding a use for the extra stress balls. It quickly became apparent that you could fling these at other team members without fear of hurting more than someone’s pride. It quickly became a tradition that the programming team has kept alive. It’s an awesome, lighthearted way to cap off a week of tough, dedicated work!

So, if you’re wandering through the office late on Friday afternoon, you’d be well advised to duck first, then grab a handful of stress balls and start throwing!

If you’re looking for a great place to buy a bunch of these things… we’ve used this company before.

Podcasting with Excellence

Being a webmonkey, I have a different take on podcasting. The Dave Ramsey Show has gone from 200 to 350 radio stations in the last 3 years due to our incredible affiliate relations/syndication team. We have Matt Aaron (spoke at NAB in ’07), one of the Industry’s best engineers running a cutting edge studio. All we do is work on the feed and the mp3s.

Amazingly, that still leaves plenty to do to get your podcast onto people’s MP3 players. The iTunes store in particular is an interesting animal. It’s perhaps the largest directory of free podcasts. Apple publishes a basic spec, but after 3 years of working at it, we’re still learning how to improve.

A few weeks ago, the iTunes store featured a personal finance section on their podcasting page. During the peak of this period, our podcast rose to #2 overall (that I saw). For a brief period of time we passed This American Life, (a podcast that’s spent a lot of time in the #1 overall slot) but were then unseated by something relating to the season finale of American Idol. Mileage may vary, but for us that exposure roughly translated into an extra 3,000 subscribers per day.

Our team culture here is to look at everything we do here in terms of lives changed. Every book, every CD, every class membership, and every advertisement sold isn’t just a number on a profit and loss statement. To us, victory is the single mom across the street and the struggling family across the country who flipped the page on their calendar knowing that for the first time in a long time they had more money than month.

To us, an extra 3,000 changed lives is worth going after.