One option would be to try to use the Usage Statistics Reporter ("USR" or "Communication Reporter"). As of SP4R2, we're still using the Poet FastObjects ("StreamServe Object Store") Querying data directly from the USR Repository would be completely and totally unsupported. But you could try editing the template project to generate an XML Consolidation Report when some other application calls for it. And then that other application could parse the XML.
Since the usage reporting is built into the StreamServer, it'll run faster, and with less overhead than anything you could script. It may not have all the functionality you want, though. (i.e. info about Collector, adjustable log level/severity filter, ability to log custom fields [like doctypes and metadata], log custom text fields...). Put together a list of your requirements, and post an Enhancement Suggestion on the PM site.
Another option would be to use the database logging introduced in SP4. In Control Center, you go to Log Configuration, enable "Database logging" and set the log level. (Refer to each application's logmanager.xml file.) Such logging happens on a per-territory basis, so the log files are stored in the Runtime Repository. You could use direct SQL calls, or Web Services calls through Service Gateway. Unfortunately, I don't have any documentation for doing this.
This approach requires the least amount of scripting, wouldn't slow down the core Strs processing as much as a fully custom solution, keeps your projects from getting too weird, uses standard logging (so you don't have to keep track of two separate sets of log messages). The existing log() scripting command is easy to use and understand; you've been using it for years. Default queries against the database log can be performed with date (including year), time (down to milliseconds), Job ID, Thread ID, log level, error code, error texts... Insert your own texts as you see fit. Instead of having to create your own framework, you could just focus on the queries. If you can get some more details from R&D about the necessary Web Services calls or database tables to hit, this would be the best option. And post your thoughts and enhancement suggestions to PM.
Your last option would be to somewhat reinvent the wheel. You'd want
This last option would require the most work. But you would have greater control over what gets logged, and how the data structures look. This could be a good thing, or a very bad thing.
- a usage client to run on each StreamServer instance. Ideally, this would be as small as possible. I'd write a small function file, which contains all the environment variables and connection parameters, and the function just does an HTTP POST. HTTP is relatively lighweight, and allows you to keep the client as simple as possible.
- a usage server to collect all the data from all the StreamServer instances. This would probably be a multithreaded StreamServer instance with an HTTP IN connector. It would store centrally to a database of some sort. If you didn't want to have this component, you could integrate the logging to database functionality in your usage client. But then each client would have to write directly to the database. And that increases your projects' complexity, and would make them run a LOT slower.
- a usage repository. I'd use a relational database. You could use the filesystem, but queries are easier with a database. Besides, with a database, you don't have to worry about concurrent writes.
- a reporting server. This would probably be a custom web app which runs queries against the database server. The customer could write this in Java, Ruby, PHP...whatever.
- a reporting client. Probably a web browser.