Object collections in Storm

13Jun10

In Storm, the Store.find method runs a query and returns matching objects. Finding all accounts in the database, the equivalent of SELECT * FROM account, is simple:

result = store.find(Account)

Clauses to limit the scope of the query can be passed to find. Finding all accounts owned by Vince Offer, the equivalent of SELECT * FROM account WHERE owner = ‘Vince Offer’, is also simple:

result = store.find(Account, Account.owner == "Vince Offer")

Another way to achieve the same thing is to take the result from the first find operation and refine it:

result1 = store.find(Account)
result2 = result1.find(Account.owner == "Vince Offer")

Storm doesn’t run a query until you do something with the result, so the impact on the database is exactly the same as in the previous example. This is the simplest form of what Jonathan and I have been calling the collection pattern. The basic idea is that you start with a collection of all objects, and then refine it, until you have the collection you want. This pattern is possible using pure Storm, as shown above, but we’ve found it useful to implement objects that hide this logic and provide a more user friendly API. For example, an AccountCollection would provide filtering logic as named methods:

class AccountCollection(object):

    def owned_by(self, salesman):
        """
        Get a new collection with accounts owned by
        salesman.
        """

    def has_unpaid_invoices(self):
        """
        Get a new collection with accounts that have
        unpaid invoices.
        """

    def find(self):
        """Get a result set of matching accounts."""

Each filtering method, such as owned_by and has_unpaid_invoices returns a new AccountCollection instance. This isn’t strictly necessary but creating a new instance is both easier to understand and implement. Finding all accounts owned by Vince Offer that have unpaid invoices is quite easy:

collection1 = AccountCollection()
collection2 = collection1.owned_by("Vince Offer")
collection3 = collection2.has_unpaid_invoices()
result = collection3.find()

This pattern provides several benefits:

  • It names filtering options which makes them easy to understand and use.
  • Query building logic is in one place which eliminates duplication.
  • All the criteria used to build a query are available when the query is generated, which makes it possible to generate optimized queries. All users of the collection benefit from optimizations.
  • It creates a clean separation between finding data and using it. This helps keep application logic more focused, because it isn’t intermingled with the particulars of generating queries.

We’ve used this pattern to good effect in both Landscape and Launchpad. In a future post I’ll talk about some different strategies we’ve used to implement the pattern.

About these ads


3 Responses to “Object collections in Storm”

  1. 1 John O'Brien

    This is similar to something I did with storm for a project I’m working on. It was slightly different in that we wanted to limit db access and transactions to be self contained (within the finder) and so it doesn’t return Storm models to prevent unwanted db access as well.

    Here are some unit tests…

    users = admin.StorageUserFinder()
    self.assertEquals(users.all(), [])
    self.assertEquals(users.count(), 0)
    self.assertEquals(users.is_empty(), True)
    self._make_users()
    #the returning object can be reused
    self.assertEquals(len(users.all()), 5)
    self.assertEquals(users.count(), 5)
    self.assertEquals(users.is_empty(), False)
    self.assertEquals(users[4].username, “tim”)
    users.filter = “BOB”
    self.assertEquals(len(users.all()), 2)
    self.assertEquals(users[0].username, “bob”)
    self.assertEquals(users[1].username, “bobby”)
    users.filter = “juan”
    self.assertEquals(len(users.all()), 1)
    self.assertTrue(isinstance(users[0], dao.StorageUser))
    self.assertEquals(users[0].username, “juan”)
    #test slicing
    users.filter = None
    subset = users[2:4]
    self.assertEquals(len(subset), 2)
    self.assertEquals(subset[0].username, “inez”)
    self.assertEquals(subset[1].username, “juan”)

    • 2 Jamu

      When are queries run? Does the finder load all the data up-front and then do in-memory testing or does each call to users.all, users.count, etc. issue a query? You might want to look at Store.block_access and Store.unblock_access. A ConnectionBlockedError is raised if an attempt to connect to the database is made when access is blocked. This was added to Storm to prevent inadvertent database access in code that runs between requests in a web application.

      It might be something you could use to let you expose model objects via your finder, while being sure that they don’t run additional queries.

      • 3 John O'Brien

        When are queries run? Does the finder load all the data up-front and then do in-memory testing or does each call to users.all, users.count, etc. issue a query?

        No it doesn’t do the query up front, It builds the storm query and runs a query when the methods are called.

        You might want to look at Store.block_access and Store.unblock_access. A ConnectionBlockedError is raised if an attempt to connect to the database is made when access is blocked. This was added to Storm to prevent inadvertent database access in code that runs between requests in a web application.

        It might be something you could use to let you expose model objects via your finder, while being sure that they don’t run additional queries.

        Yes, in some cases we could have used this, but there were other requirements where we wanted to hide the transaction management within the data layer of code. This was no easy task, but it has paid off in the end.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Follow

Get every new post delivered to your Inbox.

%d bloggers like this: