Wednesday, July 28, 2010

Application Scale and Quartz "Where"

I don't think it is to controversial to say that Quartz Scheduler is by far the dominant Open-Source Java job scheduler. It provides fast, flexible, and extremely reliable job execution and it is embedded in just about everything out there. In the last 6 months or so we have added a new Terracotta backed version of Quartz, lots of bug fixes and a GUI for monitoring quartz when clustered with Terracotta. We have a lot more coming. The team is hard at work on what's next. My favorite feature on the Quartz 2.0 list, the one they are working on right now, is Job Location Control or what is code named "Where".

Some of the trends we are seeing in our user base include the leveraging of EC2, larger apps/user bases, demanding HA requirements and hardware farms. As a result, scaled-out architectures are becoming common place in many IT environments. Scheduling work in these large multi-node environments is becoming a part of many software developers lives. Currently, Quartz supports scale-out but with little control over how the work is distributed. While this is a good start one quickly runs into problems like assigning jobs to machines that have the processing power to the work on them at the time the job is fired. Or execute the job where the data is local. Or just perform certain jobs on certain classes of machines do to their location or by purpose.

Well that is "where" we are headed. In the next few months the Quartz guys working with the Terracotta Guys and the Ehcache guys are developing a solution to the above set of problems. We'll be giving the same flexible and reliable scaled out scheduler but adding a new level of control. We are adding the "Where"

Stay tuned...

2 comments:

  1. Good move, locality of processing is very important both as a way to properly use CPU power and to achieve higher scalability by moving processing toward data.

    Keep up the good work,
    Cheers,

    Sergio B.

    ReplyDelete
  2. Steve, trying to build a multi-tenant scheduling system for a SAAS environment. What's your take on best approach? We haven't considered Terracotta, while Quartz is almost guaranteed to be used. We have a currently implemented system which needs to be migrated to a new clustered environment, to give you the full picture

    ReplyDelete