Bibliometric Analysis in promotion decisions

My chair has challenged me to come up with metrics that go above and beyond publication number or 'impact' as measured by ISI citation index. Promotions here and elsewhere betray their own peculiar mix of culture, politics and tradition. Here are my thoughts.

The basic premise is that academic productivity ought be judged on the basis of defined principles. Boyer's categories of discovery, integration, application and teaching have been quantified by promotions committees in traditional terms such as publication volume, publication influence, patents, grants, courses taught and advising undertaken. The context for these discussions is often one of promotion or tenure in academic environments, concepts themselves with questionable external validity that are increasingly under scrutiny.

While variation in and of itself isn't a problem, not examining the variation is certainly problematic. What I'm most concerned about is decisions where the lack of analysis dampens innovation or worse, hampers scholarship.

Computing and communications advances have created novel means of visualing qualitative and quantitative relationships in real time, which in turn brings variation in promotion and tenure decisions within and between organizations into sharp relief. Unfortunately I don't have answers, but did come up with some questions for myself.

Will Google Scholar replace ISI's web of science? Should professors be blogging? The editor of the Scientist thinks so, as does the Unversity of Chicago Law School. How do you cite a blog? Should you be able to tap into the attention filter of mavens? Does doing so alter the nature of their maven role? Are Boyer's categories of scholarship congruent with maven-ness, connecting, or selling? What's the role of personomies in promotion decisions?

If the nature of scholarship is changed by how it's measured, then scholarship in a web 2.0 world has more to do with the new tools to measure it. Visualizing the role of an academic and the relationships in academia over time takes on new meaning when you can do so in real time. Why don't promotion committees use academic tag clouds? Is ISI really a snobbish technorati? Where's my touchgraph ?(You must have java for the touchgraph applet to launch - what's a touchgraph?)