Paoding 2.0.2记录

Paoding 2.0.2记录

paoding 现在在svn上的代码能够支持 自动动态装载词典，并检测词典是否发生了更新、删除。
也支持关闭自动监测(paoding.stopAutoDetecting)，而提供一个方法paoding.forceDetecting手动执行一次检测。

现在这个版本为2.0.2，但是现在不打算打成jar包和zip包。
待之后2.0.3支持简繁体、提供GBK->UTF-8;Big5->utf-8转化功能后再发包。

-------------------------------
2007-9-19：
计划变更：简体繁体从2.0去除，推迟到2.1版；2.0.3版本号留空。下一个发布版本是2.0.4-alpha.
错误观点修正：因为lucene输入的是Reader，此时已经没有编码的问题了，全部都是符合unicode规范的字符了。不管是GBK还是BIG5存储的文件转化为Reader后，就没有编码的概念了。所以庖丁不存在GBK->UTF-8的变更。
-------------------------------

2.0.3之后没有特殊原因，不会再增加新的特性或功能了。
之后便是完整测试，并持续发布2.0.4-alpha;-->2.0.4-beta;-->
被**证明**稳定后最终发布2.0.5。

之后除非有严重妨碍使用的bug，否则不再发布新版本。

2.0.5之后的版本将直接跳到2.1.0开始(如果有新特性需要加入才会生版本)。
-------------------------------
2007-9-19：
计划调整：简繁体计划从2.1开始开发
-------------------------------



一个使用手动检测词典变化的例子：

	public static void main(String[] args) throws Exception {
		Paoding paoding = PaodingMaker.make();
		paoding.stopAutoDetecting();//关闭自动词典监测，使用手动检测
		PaodingAnalyzer analyzer = PaodingAnalyzer.defaultMode(paoding);
		int count = 1;
		while (true) {
			paoding.forceDetecting();//分词之前手动强制检测一次
			TokenStream ts = analyzer.tokenStream(
					"", new StringReader("庖丁解牛词典检测"));
			Token token;
			while ((token = ts.next()) != null) {
				System.out.println(token);
			}
			System.out.println("--" + (count ++) + "--");
			Thread.sleep(1000 * 5);
		}
	}

如果要使用自动监测，应该保证有其他线程在运行，否则自动监测没办法进行
(其他线程如果不存在了，那么Paoding自动退出检测，所以一般只能在Web应用中测试Paoding的自动监测)
如果检测到词典变话，可以从日志/控制台中得到消息提示。

评论

4 楼 liang1022 2008-11-13  

可以介紹一下Paoding的 開發 環 境 嗎 ?

3 楼 ylangin 2008-04-11  

请教一个分词的问题：

有一段文章，中间含“第七十四军”的文字，客户端尝试搜索“七十四军”，没有结果，
再尝试用paoding带的分词工具分，结果如下：
paoding> 第七十四军;
1:      第七/第七十/4/军/
        分词器net.paoding.analysis.analyzer.PaodingAnalyzer
        内容长度 5字符， 分 4个词
        分词耗时 31ms
--------------------------------------------------
paoding> 七十四军;
1:      74/军/
        分词器net.paoding.analysis.analyzer.PaodingAnalyzer
        内容长度 4字符， 分 2个词
        分词耗时 0ms
--------------------------------------------------
paoding>

建索引的时候送进去的文字是“第七十四军”， 这样搜索时搜索“七十四军”就没有结果了。
不知道有没有什么好的办法？

有两点考虑：
1. 所有的数字串是不是应该单独分出来，不管前面有没有修饰，比如“第”是个修饰；
2. 分次应该是“稳定”的，我的意思是，句子和句子的一部分分词的效果，对句子的一部分来讲是一样的。比如“第七十四军”和“七十四军”，“七十四军”分出“74，军”，“第七十四军”也应该分出这两个词。

谢谢。

2 楼 unkin 2007-10-25  

大侠,现在有一个问题.
在一台内网服务器paoding运行良好,在外网服务器paoding一初始化就没完,到最后一直搞到内存溢出.现贴出打印记录,请指教小子.

Oct 25, 2007 3:34:01 PM net.paoding.analysis.knife.PaodingMaker getProperties
INFO: config paoding analysis from: /home/resin-ee-2.1.16/file:/home/httpd/search/WEB-INF/lib/paoding.jar!/paoding-analysis.properties;/home/resin-ee-2.1.16/file:/home/httpd/search/WEB-INF/lib/paoding.jar!/paoding-analysis-default.properties;/home/resin-ee-2.1.16/file:/home/httpd/search/WEB-INF/lib/paoding.jar!/paoding-analyzer.properties;/home/resin-ee-2.1.16/file:/home/httpd/search/WEB-INF/lib/paoding.jar!/paoding-dic-home.properties;/home/resin-ee-2.1.16/file:/home/httpd/search/WEB-INF/lib/paoding.jar!/paoding-dic-names.properties;/home/resin-ee-2.1.16/file:/home/httpd/search/WEB-INF/lib/paoding.jar!/paoding-knives.properties;/home/resin-ee-2.1.16/file:/home/httpd/search/WEB-INF/lib/paoding.jar!/paoding-knives-user.properties
Oct 25, 2007 3:34:01 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:01 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:01 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:01 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:01 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:01 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:01 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:01 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:01 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:01 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:01 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:01 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:01 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:01 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:01 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:01 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:01 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:01 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:01 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:01 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:01 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:01 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:01 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:01 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:01 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.PaodingMaker getProperties
INFO: config paoding analysis from: /home/resin-ee-2.1.16/file:/home/httpd/search/WEB-INF/lib/paoding.jar!/paoding-analysis.properties;/home/resin-ee-2.1.16/file:/home/httpd/search/WEB-INF/lib/paoding.jar!/paoding-analysis-default.properties;/home/resin-ee-2.1.16/file:/home/httpd/search/WEB-INF/lib/paoding.jar!/paoding-analyzer.properties;/home/resin-ee-2.1.16/file:/home/httpd/search/WEB-INF/lib/paoding.jar!/paoding-dic-home.properties;/home/resin-ee-2.1.16/file:/home/httpd/search/WEB-INF/lib/paoding.jar!/paoding-dic-names.properties;/home/resin-ee-2.1.16/file:/home/httpd/search/WEB-INF/lib/paoding.jar!/paoding-knives.properties;/home/resin-ee-2.1.16/file:/home/httpd/search/WEB-INF/lib/paoding.jar!/paoding-knives-user.properties
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.PaodingMaker getProperties
INFO: config paoding analysis from: /home/resin-ee-2.1.16/file:/home/httpd/search/WEB-INF/lib/paoding.jar!/paoding-analysis.properties;/home/resin-ee-2.1.16/file:/home/httpd/search/WEB-INF/lib/paoding.jar!/paoding-analysis-default.properties;/home/resin-ee-2.1.16/file:/home/httpd/search/WEB-INF/lib/paoding.jar!/paoding-analyzer.properties;/home/resin-ee-2.1.16/file:/home/httpd/search/WEB-INF/lib/paoding.jar!/paoding-dic-home.properties;/home/resin-ee-2.1.16/file:/home/httpd/search/WEB-INF/lib/paoding.jar!/paoding-dic-names.properties;/home/resin-ee-2.1.16/file:/home/httpd/search/WEB-INF/lib/paoding.jar!/paoding-knives.properties;/home/resin-ee-2.1.16/file:/home/httpd/search/WEB-INF/lib/paoding.jar!/paoding-knives-user.properties
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:02 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:03 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:04 PM net.paoding.analysis.knife.PaodingMaker getProperties
INFO: config paoding analysis from: /home/resin-ee-2.1.16/file:/home/httpd/search/WEB-INF/lib/paoding.jar!/paoding-analysis.properties;/home/resin-ee-2.1.16/file:/home/httpd/search/WEB-INF/lib/paoding.jar!/paoding-analysis-default.properties;/home/resin-ee-2.1.16/file:/home/httpd/search/WEB-INF/lib/paoding.jar!/paoding-analyzer.properties;/home/resin-ee-2.1.16/file:/home/httpd/search/WEB-INF/lib/paoding.jar!/paoding-dic-home.properties;/home/resin-ee-2.1.16/file:/home/httpd/search/WEB-INF/lib/paoding.jar!/paoding-dic-names.properties;/home/resin-ee-2.1.16/file:/home/httpd/search/WEB-INF/lib/paoding.jar!/paoding-knives.properties;/home/resin-ee-2.1.16/file:/home/httpd/search/WEB-INF/lib/paoding.jar!/paoding-knives-user.properties
Oct 25, 2007 3:34:04 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:06 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:06 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:07 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:07 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:07 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:07 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:09 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:10 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:10 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:14 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:14 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:34:22 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
java.lang.OutOfMemoryError: Java heap space
Oct 25, 2007 3:36:00 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
Oct 25, 2007 3:36:00 PM net.paoding.analysis.knife.FileDictionaries loadAllWordsIfNecessary
INFO: loading dictionaries from /home/httpd/search/WEB-INF/classes/dic
java.lang.OutOfMemoryError: Java heap space

1 楼 shguan 2007-09-11  

GBK->UTF-8

可以借助 Apache Commons-IO 项目中提供的实用工具来编写代码。
    /* gbkString 为一 GBK 编码的字符串 */
    String utf8String = IOUtils.toString(IOUtils.toInputStream(gbkString, "UTF-8"));
utf8String 中字符，皆变为 UTF-8 编码。

附，com.apache.commons.io.IOUtils 中相关代码如下：
    /**
     * Convert the specified string to an input stream, encoded as bytes
     * using the specified character encoding.
     * <p>
     * Character encoding names can be found at
     * <a href="http://www.iana.org/assignments/character-sets">IANA</a>.
     *
     * @param input the string to convert
     * @param encoding the encoding to use, null means platform default
     * @throws IOException if the encoding is invalid
     * @return an input stream
     * @since Commons IO 1.1
     */
    public static InputStream toInputStream(String input, String encoding) throws IOException {
        byte[] bytes = encoding != null ? input.getBytes(encoding) : input.getBytes();
        return new ByteArrayInputStream(bytes);
    }