Performance Architecture

I was reading this article from another guy named Steve Harris (not me). I think he makes some interesting points but I thought I would follow up. Optimizing architecture to me mostly means designing for flexibility and testability. It also means having a way to test ones app for performance and scale. I've seen a number of people write apps, put them in prod without ever performance testing them.

Write your app so that you can overlay the performance architecture after the fact. Have a way to EASILY validate your performance and you'll be ok. Optimizing code early (which he doesn't mention) is actually quite evil and almost always leads to badness.

Teams, starts with the people

What are the needed traits across a company to have the greatest chance of success? How much of each trait does each individual have to have? These are questions that everyone should ask themselves on both sides of the interview process and in ones everyday life.

My list is as follows (not in any order):

• Passion/Pride/Work ethic
  
• Knowledge/experience
• Respect/Teamwork
  
• Judgement
• Intelligence

The list itself can vary though probably not as widely as how much one values each item on that list. The most important thing is to have the list and to use it as a framework for how you choose jobs, hire, value and motivate existing people, and at times part ways.

Since words tend to have very different meanings to different people despite the existence of dictionaries and wiki's I'm going to cover a bit what my list means to me.

Passion/Pride/Work Ethic

First let me start with what this doesn't mean. Passion and pride does not have anything to do with outward energy. One does not have to be "Rah Rah" to have passion and pride. Many of the people who I feel have had the most passion have actually been super quiet. Passion is about commitment to the task at hand and the greater goal. It's about going the extra mile to get things done well, on time, and sometimes in ways that nobody else thought of. It's about finding the things that need to get done that nobody noticed and being the person to do them while not dropping the things they are supposed to be doing :-).

Knowledge

Knowledge is what you "know" coming into a problem. Intelligence is what you can learn/how well one solves problems. In many cases knowledge is the least important of the five. For given positions one should always give some thought to how much specific knowledge is actually needed. I constantly see job postings with a long list of knowledge items and almost no indication of the other 4 items importance. IMHO If you can find people who excel in the other four areas, knowledge of a given specific task can be lower (though not zero. Working with people who have zero knowledge of their domain can be very expensive. It's less of an issue for large companies with time and money).

Respect/teamwork

I focus on respect more than one might imagine. I worked with a guy who said he spends 75 percent of his time in an interview trying to figure out if the person will work well in the team. I think that's a good idea. It's not about whether the person is social or fun. It's about whether they present their ideas with an understanding that those around may have better ones. Do they listen to feedback, even from people they do not expect to have good ideas. Do they listen in general or do they just wait for their turn to talk. This is important stuff. Don't underestimate it.

Judgement

This is another really important one. Their is a nice coverage of how important this is in the book "Producing Open Source Software.(A free version of this book exists)" Whether someone is your boss, your co-worker or your employee, you need to be able to trust their judgement more than anything else. If the people around you are smart but have bad judgement they will create excellent solutions to the wrong problems. If people are not smart and have bad judgement they will create poor solutions to the wrong problems. On the other side of the coin, if they are smart and have good judgement they will create great solutions to the right problems. Most importantly, if they are not smart (and on any given day all of us are not smart) but have excellent judgement they will still make good decisions about what to do, and most usefully, what not to do. While I find it really really hard to interview for, Judgement is huge!

Intelligence

Having some Intelligence is important. More important for some jobs than for others. When hiring, know how much intelligence (balanced against the others) is needed for the position/type of work. For some jobs, judgement and hard work with some intelligence is what is needed. Have questions in mind that will indicate whether a person is smart enough. Since intelligence is the easiest of the bunch to test for Don't Forget to Do it! To many people mistake liking someone, "talking the talk" or even looks for intelligence. Do the work, have your intelligence questions in mind when you interview someone.

The above are some high order bits. When I again get some time and motivation I want to cover team building from a balance perspective. Recognizing and filling ones personal and team gaps.

Ok, it's 2:30 am and I'm starting to question my judgement for being up so late. What's my point here? Think about what you want in employees coworkers and bosses. Sometimes even assign numbers to the core values you look for so you can compare people.

I read that we make up our mind about people we meet in seconds. Before they even speak in most cases and that those first impressions are VERY hard to overcome. So have the tools ready to make wise informed decisions and fight past human nature.

The int size problem...

I'm just curious. Of the people out there using 64bit jvms, has anyone run into the int collection size problem? One of the things I've been quitely worried about for awhile now is that with both 64 bit jvm's and the virtual heap stuff provided by Terracotta DSO it is possible for an ArrayList or a HashMap to grow beyond what an integer can hold. Sure 2 gig of objects is a ton to be in a collection but if one has a 128g heap it isn't crazy anymore.
It's easy to make the collections able to handle the larger sizes but so much code is written to the current collection interfaces it might be a problem. Have the smart people at Sun started tackling this already? Maybe some JSR?

Programming by Wishful Thinking

I'm currently reading an excellent book called Everyday Scripting. I'm really enjoying it as a nice primer on whipping things up with Ruby. When I hit chapter 7 the author began to discuss a programming style that I have used for a while now. I had no idea this simple technique had a name. Many of the things I've picked up throughout my career I've learned from books, articles, reading code, and still others I've devised from first principals. This particular practice I picked up while pair programming with this super smart guy Peter Seibel. Apparently it's called "Programming by Wishful thinking."

The idea is that when you have some code to write you first do so as if methods existed already to perform the high level operations. Say you want to write some code that tells you how many songs and albums you have by a given artist. Programming languages don't have methods do to this baked in but you pretend they do. An extreme example of the practice would be:



I came up with this example in about thirty seconds so I hope it isn't to opaque.

You may then go on to write the above methods in the same way:



and so on and so on. Anyway, don't know how many people out there use this style but I've found it really helpful when I need to keep focused on the task/flow at hand without getting to distracted by the details of an issue. Maybe it will help someone else too.

Wisdom and Terracotta... Good mix

This guy wrote about two topics that I like. The book wisdom of crowds and Terracotta DSO:

http://blogs.techrepublic.com.com/opensource/?p=31

Pairing can make you stupid...

Pretty much everyone has heard of or tried pair programming. I personally have found it to be quite valuable. One very interesting thing I was reminded of when watching some people on my team do some pairing is how dumb pair programming makes the typer. Conversely, it also makes the watcher (don't know what it's really called) considerably smarter.

I can vividly remember pairing with someone much smarter than me on a transaction logging system a few years back and noticing this effect. When he was typing, I was always coming up with the cool ideas. Of course it's not always an intelligence thing. Around the office I have noticed multiple pairs demonstrate this. The typer becomes dump. The non-typers make excellent judgments and decisions.

I think the obvious explanation for this is just how much energy we put into things like typing, picking variable names, formatting, etc... I've often joked that we should hire secretaries with a little programming knowledge for each programmer.

Has anyone else out there noticed this? Is their anything to be gained from this observation? Or am I completely off base.

Performance Tuning with Thread Dumps

As a follow on to my earlier tuning blog I whipped up a quick sample to make the Performance tuning with Thread dumps point clearer (see my previous blog). Take a look at the following code and make a guess at what you think the slowest part of the code is and why.

package softwareandstuff.performance;

import java.util.ArrayList;
import java.util.List;
import java.util.Random;

public class SomethingToRun {

 public void run() {
   List stuff = new ArrayList();
   Random r = new Random();
   long total = 0;
   long count = 0;
   while (true) {
     long start = System.currentTimeMillis();
     String s = " " + r.nextInt() + " " + r.nextInt();

     for (int i = 0; i < i =" 0;" 1000 ="="">

On my slow computer it took around 43 seconds per 1000.

Now use jps to find out what process id the little app has and do a "kill -3 " 10 to 20 times fast in succession.

Note: jps requires jdk1.5 to be installed. Figure out how to use task manager in windows or ps on unix based platforms if you are not on jdk1.5

steve:~ steve$  jps

7595 Jps
7594 SomethingToRun
7440 startup.jar

steve:~ steve$ kill -3 7594

Note: See the following link for more on how to do thread dumps and what they are A little on thread dumps

Should see something like the following:

Signal Dispatcher" daemon prio=9 tid=0x005098d0 nid=0x1820a00 waiting on condition [0x00000000..0x00000000]

"Finalizer" daemon prio=8 tid=0x00509030 nid=0x180fc00 in Object.wait() [0xf0a04000..0xf0a04ab0]
at java.lang.Object.wait(Native Method)
- waiting on <0x266101e0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
- locked <0x266101e0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x00508c40 nid=0x180ec00 in Object.wait() [0xf0983000..0xf0983ab0]
at java.lang.Object.wait(Native Method)
- waiting on <0x26610260> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:474)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x26610260> (a java.lang.ref.Reference$Lock)

"main" prio=5 tid=0x005013b0 nid=0x1804600 runnable [0xf07ff000..0xf08002c0]
at softwareandstuff.performance.SomethingToRun.run(SomethingToRun.java:25)
at softwareandstuff.performance.SomethingToRun.main(SomethingToRun.java:36)

"VM Thread" prio=9 tid=0x00508460 nid=0x1803c00 runnable

Note: ignore any thread that is in Object.wait() for now. In fact, searching on runnable is usually your best bet for skipping to the things that are interesting from a performance perspective.

My guess is what you will see is that a decent percentage of the time is spent in the remove call. hmmm, wonder why that is. Now go look at the implementation of remove on an ArrayList and it will all be clear. Now try using a linked list instead. For me that dropped it down to 6 seconds per 1000.

busy today but more to come...