This isn't a full solution, but you may want to look into OrientDB as part of your stack. Orient is a Graph-Document database server written entirely in Java.
In graph databases, relationships are considered first class citizens and therefore traversing those relationships can be done pretty quickly. Orient is also a document database which would allow you the kind of schema-free architecture it sounds like you would need. The real reason I suggest Orient, however, is because of its extensiblity. It supports streaming via sockets, and the entire database can be embedded into another application. Finally, it can be scaled efficiently and/or can work entirely through memory. So, with some Java expertise, you can actually run your preset queries against the database in memory.
We are doing something similar. In creating an app/site for social science research collaboration, we found ourselves with immensely complex data models. We ended up writing several of the queries using the Gremlin Traversal Language (a subset of Groovy, which is, of course, Java at its heart), and then exposing those queries through the binary connection server of the OrientDB. So, the client opens a TCP socket, sends a short binary message, and the query is executing in Java directly against the in-memory database.
For something of this size, I would want to use Orient in conjunction with Hadoop or something like that. You can also use Orient in conjunction with esper.
An introduction to orient: http://www.sitepoint.com/a-look-at-orientdb-the-graph-document-nosql/
Complex, real-time queries: http://www.gft-blog.com/business-trends/leveraging-real-time-scoring-through-bigdata-to-detect-insurance-fraud/
A discussion about streaming options with java and orient: https://github.com/orientechnologies/orientdb/issues/1227