Author Archive

Treating New Team Members Well

December 14, 2011

One of the great things about working for Dave Ramsey at the Lampo Group is that they treat all team members equally well. This is especially true of the Web Development Team, which has its own flavor of generosity.

One of the Web Development traditions is to help new team members move. When I was hired, I was told that if needed, 5-10 guys would show up to help my family move in, which we gratefully accepted. When we moved in, which was a Monday afternoon no less, a total of 6 or 7 guys showed up, most of which were not on my immediate team but on our Endorsed Local Providers team. One guy even worked an earlier shift from 7:30-4:30 just to come help, and the rest showed up shortly after 5:30. We had all of our items moved in to our townhome in under 2 hours.

Since joining the team, I have joined this tradition by helping two other families move in. Both chose to move on the Saturday immediately before their first day of work, and both times we were able to move them in within 2-3 hours. They were amazed at how quickly we were able to get them in, and it allowed them to have a little more family time before starting work that following Monday. It is probably even more amazing to their spouses to see just how much effort the team is willing to put in for someone so new.

Last week something new happened. A new developer was supposed to join my team, but could not show up on his first day due to a medical condition that came up while he and his family were moving. For one reason or another, they had professional help moving them, so no moving team was assembled. It was not until the Sunday before he was to start that we knew he was going to be out. Since we all had to be at work during the week, several of the team members’ wives jumped into action. They cooked meals for this new family, helped his wife unpack and get into their house, and in general made this family feel welcome.

It’s not everyday that you get to work for an employer, a team, and with families that care so much about all team members, new or old.

What does your company do to help new team members?

We have a wonderful team here at The Lampo Group, and are always looking for more talented developers.

Advertisements

Performance of CFScript

September 9, 2011

I have been developing ColdFusion on and off for about 10 years, and one topic that has been controversial the entire time is CFScript. The arguments go back and forth: ‘It’s too slow’, ‘it’s not ColdFusion’, ‘It looks like Javascript, but isn’t’.

On our team, the majority of our code is written in plain CFML. With new team members coming onboard with stronger backgrounds in Java-like languages, more and more CFScript is starting to show up, especially in the business-tier. The question of performance has come up, and though we do not have real numbers to back it up, the general consensus is that the performance difference is negligible, and each team can choose if they want to use CFML or CFScript going forward.

Not satisfied with a general feeling, I set out to conduct a very simple test, and see what ColdFusion does behind the scenes when compiling a simple component in CFML, and the same component written in CFScript. The component I came up with has a single function, with a single numeric argument. It then loops a variable number of times, constructs a struct, and does some simple math.

First, the CFML version. I added a suppressWhitespace=”true” directive around the component, in addition to output=”false”, to be completely sure that whitespace would not be emitted.

<cfcomponent output="false">
<cfprocessingdirective suppressWhitespace="true">
	<cffunction access="public" name="doStuff" returntype="void"  output="false">
		<cfargument name="iterations" type="numeric" required="true" />
		<cfloop from="1" to="#arguments.iterations#" index="local.i">
			<cfset local.myStruct = {a=1,b=local.i}/>
			<cfset local.myStruct.c = local.myStruct.a*local.myStruct.b/>
		</cfloop>
	</cffunction>
</cfprocessingdirective>
</cfcomponent>

Next up, CFScript. Notice there is no output=’false’ or suppressWhitespace equivalent. As I will show in a moment, CFScript components do not emit whitespace.

component {
	public void function doStuff(required numeric iterations){
		for (local.i = 1; local.i <= arguments.iterations; local.i++) {
			local.myStruct = {a=1,b=local.i};
			local.myStruct.c = local.myStruct.a*local.myStruct.b;
		}
	}
}

I turned on ‘Component cache’ and ‘Save class files’ in the ColdFusion Administrator, and ran the resulting .class files through the JAD decompiler.

The results of the decompilation don’t look the greatest here on WordPress, so here are links to the resulting decompiled Java classes:

A few observations:

  • The CFML version produces a runPage() method that has a lot of logic dealing with the cfprocessingdirective tag
  • Even with output=false and suppresswhitespace, the CFML version has several calls of the format _whitespace(out, “\r\n\t\t\t”);. With the suppresswhitespace in place, this still results in several function calls in the ColdFusion runtime, going from CfJspPage, to NeoPageContext, to CFOutput, before finally deciding to throw it away.
  • CFScript has no such whitespace method calls, so adding output=’false’ to CFScript is not necessary.
  • The CFML and CFScript versions of the FOR loop compile quite differently. The CFML version is somewhat tame at ‘for (; CfJspPage._checkCondition(t11, t15, t13); CfJspPage.checkRequestTimeout(“CFLOOP”))’, while the CFSCRIPT version is a little more complex, doing bit shifting in one place and a few extra casts.

Other than these differences, the compiled code is almost identical.

I also used JMeter to run simulated usage across the two components. With 8 simulated users across 4000 calls, each calling both CFCs in a random order and each passing 10,000 to the component (resulting in 40m total iterations), both averaged around 116ms per call, and consistently came within 3-4ms of each other.

This is far from an exhaustive comparison of CFML and CFScript. There are things that CFML can do that either you cannot do at all in CFScript, or have to jump through some hoops to do. Both flavors of the language have their places, and as for business-layer code that can be written without jumping through hoops, CFScript performs just as well as CFML.

Logging in to Salesforce from ColdFusion

August 31, 2011

We have been working the last few months on an experiment with Salesforce.com. The code is some of our first to be written almost purely using CFScript and Hibernate. One of our challenges was communicating with Salesforce, in particular, their fairly new REST API. Their API documentation is pretty good, but the one area that was somewhat lacking at the time was how to log in. Once we figured it out, the solution was rather simple, it just took a little while to find the right combination of parameters.

We hope to eventually release our entire CFC for talking with Salesforce, which has a lot of convenience methods to get, set, upsert, and describe Salesforce objects, but for now, here is our code to log in and get an object.

public boolean function logIn() {
	if (variables.token EQ '' OR arguments.force) {
		local.http = new Http(url=variables.salesforceInstance & '/services/oauth2/token',method='post');
		local.http.addParam(type='formField',name='grant_type',		value='password');
		local.http.addParam(type='formField',name='client_id',		value=variables.CLIENTID);
		local.http.addParam(type='formField',name='client_secret',	value=variables.CLIENTSECRET);
		local.http.addParam(type='formField',name='username',		value=variables.USERNAME);
		local.http.addParam(type='formField',name='password',		value=variables.PASSWORD);
		local.http.addParam(type='formField',name='format',		value='json');

		local.httpSendResult = local.http.send();
		local.httpResult = httpSendResult.getPrefix();

		if (StructKeyExists(local.httpResult, "responseHeader")
				&& StructKeyExists(local.httpResult.responseHeader, "status_code")
				&& local.httpResult.responseHeader.status_code EQ 200) {
			local.stringResult = gatherResponseString(local.httpResult);
			//variables.json is an in-house wrapper around 'org.json.simple.JSONValue'. DeserializeJSON may work for you as well
			local.json = variables.json.toObject(local.stringResult);
			variables.token = local.json.access_token;
		} else {
			variables.token = '';
			local.errorString = gatherResponseString(local.httpResult);
			if (StructKeyExists(local.httpResult, "ErrorDetail")) {
				local.errorString &= ", detail: " & local.httpResult.ErrorDetail;
			}
			throw (message='Unable to authenticate to SalesForce: ' & local.errorString, type='lampo.salesforce.loginerror');
		}
	}
}
public any function getObject(required string sfid, required string type, array fields) {
	local.url=variables.salesforceInstance & '/services/data/v' & variables.apiVersion &'/sobjects/'&arguments.type&'/' & arguments.sfid;

	if (NOT isNull(fields) AND ArrayLen(fields) GT 0) {
		local.url = local.url & '?fields=' & ArrayToList(fields);
	}

	return makeJsonGetCall(url=local.url,errorMessage='Unable to find ' & arguments.type & ' with sfid ' & arguments.sfid);
}

/*************** HELPER FUNCTIONS ********************/
private function gatherResponseString(required any httpresult) {
	if (IsBinary(httpResult.FileContent)) {
		return CharsetEncode(httpResult.FileContent, httpResult.CharSet);
	} else {
		return trim(httpResult.FileContent);
	}
}

private any function makeJsonGetCall(required string url, required string errorMessage) {
	local.httpResult = makeCall(arguments.url,'GET',{});

	if (local.httpResult.responseHeader.status_code EQ 200) {
		local.fromJson = variables.json.toObject(gatherResponseString(local.httpResult));
		return local.fromJson;
	} else {
		throw (message=arguments.errorMessage & ' (status: '& local.httpResult.responseHeader.status_code &')',type='lampo.salesforce.objectNotFound')	;
	}
}
private any function makeCall(required string url, required string method,required struct params) {
	local.http = new Http(url=arguments.url,method=arguments.method);
	for (paramType in params) {
		for (paramKey in params[paramType]) {
			if (paramType NEQ 'header' or paramType neq 'Authorization') {
				local.http.addParam(type=paramType,name=paramKey,value=params[paramType][paramKey]);
			}
		}
	}
	local.http.addParam(type='header',name='Authorization',value='OAuth ' & variables.token);
	local.httpSendResult = local.http.send().getPrefix();

	if (local.httpSendResult.responseHeader.status_code EQ 401) {
		throw (message = 'Unable to log into SalesForce: ' & local.httpSendResult.responseHeader.status_code & '('&gatherResponseString(local.httpSendResult)&')',detail=gatherResponseString(local.httpSendResult),type='lampo.salesforce.loginFailure');
	} else {
		return local.httpSendResult;
	}
}

Coldfusion 9 ORM, Caching and Autocommit

August 22, 2011

We have been using ColdFusion 9 for a few months now. With all new code that is developed, we have been abandoning <CFQUERY> in favor of CF9’s ORM, which is based on Hibernate. This is a great technology that lets us focus more on work that matters, instead of coding lots of repetitive SQL queries. But, as with any new technology, there are some gotchas that developers have to be aware of. Two such gotchas have come up recently on projects I have worked on, and I am going to share those with you today.

Setup

For this simple example, we are going to use a very basic table.

create table dbo.person (
	[id] [int] IDENTITY(1,1) NOT NULL,
	[first_name] [varchar](128) NOT NULL,
	[last_name] [varchar](128) NOT NULL,
);

In traditional ColdFusion, if you wanted to update a person, you would perform a with an update statement, like so:

update person set first_name=
where id =

In ColdFusion 9 with ORM, you have an object that represents a person, as defined in Person.cfc. Note that I will be using 100% CfScript for the rest of this article.

component persistent="true" table="person" {
	property name="id" fieldtype="id" generator="native";
	property name="firstName" column="first_name";
	property name="lastName" column="last_name";
}

To update the person, you would write something like the following:

local.person = EntityLoadByPk('Person',arguments.personId);
local.person.setFirstName(arguments.newFirstName);
EntitySave(local.person);

So far, this is all basic ColdFusion ORM. Let’s look at some things you need to be aware of as your application grows in complexity.

Gotcha 1 : Caching

Consider the following MxUnit code:

public void function testFunction1() {
	local.person = EntityLoad('Person',{firstName='John',lastName='Doe'},true);
	local.person.setFirstName('Jane');
	otherFunction1();
}
private void function otherFunction1() {
	local.person = EntityLoad('Person',{firstName='John',lastName='Doe'},false);
	assertEquals(1,local.person.size());
	assertEquals('Jane',local.person[1].getFirstName());
}

In function otherFunction1, we have asked Hibernate to give us John Doe a second time. One would expect this to return a fresh copy of John Doe, but actually it has returned a reference to the same person as inside testFunction1. The query is indeed performed on ‘John Doe’, but in the scope of the current request, that maps to an object already in memory, one that has a new first name. You can read a little more about this form of caching in the CF docs.

It is important to also see what happens when you use EntityReload. Since both variables point to the same instance from the cache, an EntityReload on will will reload the other. Consider the following:

public void function testFunction2() {
	local.person = EntityLoad('Person',{firstName='John',lastName='Doe'},true);
	local.person.setFirstName('Jane');
	otherFunction2();
	assertEquals('John',local.person.getFirstName());
}
private void function otherFunction2() {
	local.person = EntityLoad('Person',{firstName='John',lastName='Doe'},true);
	assertEquals('Jane',local.person.getFirstName());
	EntityReload(local.person);
	assertEquals('John',local.person.getFirstName());
}

Gotcha 2: Autocommit

By default, Hibernate is configured to commit any changed objects. In our above example, the EntitySave(local.person) is completely optional – at the end of the request, the changes to the person will be persisted to the database. What’s bad about this is that in the pre-ORM world, developers in ColdFusion are used to explicitly updating the database via____ or some other framework. You would think that the equivalent would be calling EntitySave.

This recently bit me when I was validating some data before writing it back to the database. We have a form that is used to update contact information, and allows for our users to add new types of contacts to a person (such as billing e-mail, toll-free number, etc). There are some rules involved, one being that certain people must have one and only one billing e-mail address. Instead of validating the form and the database separately, the code I was working on converted the new contact field from the form into a new database object (via EntityNew), added it to the list of existing contact fields, and then ran the validation on the objects. If validation failed, the idea was to present the user with an error so they could correct it, and new or updated information would not make it to the datbase. Instead, Hibernate went ahead and saved all changes to the database, and allowed the error message to be displayed.

To work around this, there are a few things you can do:

  1. Don’t mess with objects unless you want your changes to always be committed
  2. Turn off autocommit by adding <property name=”connection.autocommit”>true</property> to your hibernate.xml file.
  3. If you want to back out any changes you have made to an object, call EntityReload(local.myEntity) to discard changes
  4. Use transactions
I will elaborate a little bit on #4. Assume the database has John Doe in it as you look at the following MxUnit code:
public void function testFunction3() {
	transaction {
		local.person = EntityLoad('Person',{firstName='John',lastName='Doe'},true);
		local.person.setFirstName('Jane');
		transactionrollback();
		assertEquals('Jane',local.person.getFirstName());
	}
	otherFunction3();
	assertEquals('Jane',local.person.getFirstName());
}

private void function otherFunction3() {
	transaction {
		local.person = EntityLoad('Person',{firstName='John',lastName='Doe'},false);
		assertEquals(1,local.person.size());
		assertEquals('John',local.person[1].getFirstName());
	}
}

Note that transactionRollback() does not rollback the change to the object, but it does appear to invalidate the cache. The second method gets a fresh copy from the database.

These gotchas can be very good things, but if you develop your application without being aware of them, you will be very surprised when data starts appearing in your database that you thought you had thrown away.

Salesforce and Amazon SQS

March 8, 2011

A couple of us are working on a new project that involves Salesforce CRM. We recently came upon a few technical challenges related to our need to keep a subset of the data stored in the cloud in-sync with our internal application servers. These include:

  • Limited number of API calls we can make per 24 hour period. Salesforce charges a per-user license fee, and each license gives you a certain number of API calls. If our project scales as we expect it to, we could easily exceed the number of API calls.
  • Keeping development data in sync with our internal development server and test data with the test server.

We came up with two initial solutions:

  1. Have our internal systems poll the cloud for change.
  2. Have the cloud send a message to our systems informing us of a change.

The first approach requires a balancing act: how up-to-date does the information need to be on our side, vs. how many API calls we can make. If we poll every second, we would consume 86,400 calls per day – more than we will probably have allotted when we launch.  We also can’t consume 100% of our API calls on polls, as once we have detected that something needs to sync, we need to make calls to download the changed objects, and also need calls to periodically send data to the cloud as well.

The second approach seems to be the better one, as outbound messages don’t apply towards daily API limits. Also, we only anticipate the synced object types would only ever incur a few hundred changes per day, far fewer than the thousands of polling calls we would have to make. The problem then becomes how to implement the sync in a way that would work in our production ‘org’, our ‘sandbox’, and the ‘developer accounts’ that we developers are using. The way our web stack is structured, however, only allows for communication from third parties to see our production web environment. We could come up with our own way to queue messages in production intended for other levels, but we would need to be even more concerned about security and we would likely be duplicating something that the marketplace already provides.

It turns out someone does: Amazon.

I’ve been familiar with Amazon’s cloud computing offerings for some time now, just have never been able to utilize them with previous employers. Amazon has a service known as SQS, or Simple Queue Service, that “offers a reliable, highly scalable, hosted queue for storing messages as they travel between computers.”

With SQS you can:

  • Send up to 64KB of text per message
  • Persist messages for up to 14 days
  • Create unlimited queues (ie: one queue for each of our environments)
  • And a lot more

Furthermore, SQS is cheap: $0.01 per 10,000 requests (send and receive are considered separate), and about $0.1 per GB of transfer. Far cheaper than buying additional Salesforce licenses.

Salesforce has its own language known as Apex, which runs on their servers and has a syntax very similar to Java’s. SQS messages are fairly simple, with Query and SOAP based API’s available. The one complexity is the means of signing a message. SQS messages include HMAC signatures using a private key you establish with Amazon that prevents messages from being forged.

The SQS implementation is quite simple. An Apex Trigger exists on object that we need to sync.  That trigger enqueues a message containing the record type, the ID of the changed record, and a timestamp. This message goes to a SQS queue that corresponds to the environment (dev, test, prod, etc…).  A scheduled task on our end polls SQS every few seconds for changes.

How do you sign a message in Apex that conforms to SQS specs? Apex does have some good built in libraries, including a Crypto class that even has an AWS example in their documentation (though for a different service using a much simpler authentication scheme). Here is the solution I came up with:

<pre>public class AmazonSqsSender
{

	private String getCurrentDate() {
		return DateTime.now().formatGmt('yyyy-MM-dd\'T\'HH:mm:ss.SSS\'Z\'');
	}

	public void sendMessage(String message) {
		//AmazonAws__c is a custom setting object that stores our keys, an Amazon Host, and a queue name
		//You can just put your keys, host and queue below as strings
		AmazonAws__c aws = AmazonAws__c.getOrgDefaults();

		String accessKey =aws.accessKey__c;
		String secretKey = aws.secretKey__c;
		String host = aws.host__c;
		String queue = aws.queue__c;

		Map<String,String> params = new Map<String,String>();

		params.put('AWSAccessKeyId',encode(accessKey));
		params.put('Action','SendMessage');
		params.put('MessageBody',encode(message));
		params.put('Timestamp',encode(getCurrentDate()));
		params.put('SignatureMethod','HmacSHA1');
		params.put('SignatureVersion','2');
		params.put('Version','2009-02-01');

		//The string to sign has to be sorted by keys
		List<String> sortedKeys = new List<String>();
		sortedKeys.addAll(params.keySet());
		sortedKeys.sort();

		String toSign = 'GET\n' + host +'\n'+queue+'\n';
		Integer p = 0;
		for (String key : sortedKeys) {
			String value = params.get(key);
			if (p > 0) {
				toSign += '&';
			}
			p++;
			toSign += key+'='+value;
		}
		params.put('Signature',getMac(toSign,secretKey));

		String url = 'https://'+ host+queue+'?';
		p = 0;
		for (String key : params.keySet()) {
			if (p > 0) {
				url += '&';
			}
			p++;
			url += key+'='+params.get(key);
		}

		HttpRequest req = new HttpRequest();
		req.setEndPoint(url);
		req.setMethod('GET');
		Http http = new Http();
		try {
			//System.debug('Signed string: ' + toSign);
			//System.debug('Url: ' + url);
			HttpResponse res = http.send(req);
			//System.debug('Status: ' + res.getStatus());
			//System.debug('Code  : ' + res.getStatusCode());
			//System.debug('Body  : ' + res.getBody());
		}
		catch (System.CalloutException e) {
			System.debug('ERROR: ' + e);
		}

	}
//Amazon wants + and * to be escaped, but not ~
	private String encode(String message){
		return EncodingUtil.urlEncode(message,'UTF-8').replace('+', '%20').replace('*', '%2A').replace('%7E','~');
	}

	private String getMac(String RequestString, String secretkey) {
		String algorithmName = 'hmacSHA1';
		Blob input = Blob.valueOf(RequestString);
		Blob key = Blob.valueOf(secretkey);
		Blob signing =Crypto.generateMac(algorithmName, input, key);
		return EncodingUtil.urlEncode(EncodingUtil.base64Encode(signing), 'UTF-8');
	}

	public static void sendTest() {
		AmazonSqsSender t = new AmazonSqsSender();
		t.sendMessage('Hello from Salesforce ' + Math.random());
	}
}

Using the System Log, it is possible to call AmazonSqsSender.sendTest() to send a random message. Some Java code was running on my workstation that proved messages were being sent.

For the time being, we are going to poll Salesforce directly to keep the overall complexity down, but at least we know that Amazon SQS is an option if we need it.