Web mining from competitors' websites

This paper presents a framework for user-oriented text mining. It is then illustrated with an example of discovering knowledge from competitors' websites. The knowledge to be discovered is in the form of association rules. A user's background knowledge is represented as a concept hierarchy developed from documents on his/her own website. The concept hierarchy captures the semantic usage of words and relationships among words in background documents. Association rules are identified among the noun phrases extracted from documents on competitors' websites. The interestingness measure, i.e. novelty, which measures the semantic distance between the antecedent and the consequent of a rule in the background knowledge, is computed from the co-occurrence frequency of words and the connection lengths among words in the concept hierarchy. A user evaluation of the novelty of discovered rules demonstrates that the correlation between the algorithm and the human judges is comparable to that between human judges.