這個(gè)是發(fā)生在上周周末的真實(shí)案例，因?yàn)閏xf client 端線程安全導(dǎo)致的錯(cuò)誤，總結(jié)出來(lái)希望其他使用cxf的兄弟注意。
首先描述一下背景，簡(jiǎn)單的說(shuō)就是使用cxf作為web service的客戶端，運(yùn)行在weblogic上，連接外部的服務(wù)器。為了測(cè)試需要，開(kāi)發(fā)了一個(gè)簡(jiǎn)單的模擬器模擬服務(wù)器端，準(zhǔn)備在release之前跑穩(wěn)定性測(cè)試。
結(jié)果出問(wèn)題了，在排除掉一些干擾和諸如網(wǎng)絡(luò)環(huán)境，設(shè)置等之后問(wèn)題依舊，由于系統(tǒng)負(fù)責(zé)，包括ws的模擬器也是出了一個(gè)之前沒(méi)有試過(guò)的方法，因此費(fèi)了不少時(shí)間來(lái)查找問(wèn)題。過(guò)程很枯燥，應(yīng)該很多人經(jīng)歷過(guò)，在一個(gè)大的系統(tǒng)中找到一個(gè)小錯(cuò)誤的出處，可以說(shuō)是一門學(xué)問(wèn)，技術(shù)耐心和運(yùn)氣都是需要的.....跳出這個(gè)過(guò)程，由于問(wèn)題表現(xiàn)在web service的網(wǎng)絡(luò)連接在這個(gè)異常上，在服務(wù)器端模擬器的日志中有大量的這種異常信息：
2009-07-24 19:23:22,898 DEBUG ( : ) (tomcat-exec-56) [Http11NioProcessor] - Error parsing HTTP request header
java.io.EOFException: Unexpected EOF read on the socket
at org.apache.coyote.http11.InternalNioInputBuffer.readSocket(InternalNioInputBuffer.java:589)
at org.apache.coyote.http11.InternalNioInputBuffer.parseRequestLine(InternalNioInputBuffer.java:425)
at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:825)
at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:719)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2080)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
2009-07-24 19:23:22,898 DEBUG ( : ) (tomcat-exec-56) [Http11NioProcessor] - Error parsing HTTP request header
java.io.EOFException: Unexpected EOF read on the socket
at org.apache.coyote.http11.InternalNioInputBuffer.readSocket(InternalNioInputBuffer.java:589)
at org.apache.coyote.http11.InternalNioInputBuffer.parseRequestLine(InternalNioInputBuffer.java:425)
at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:825)
at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:719)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2080)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
而服務(wù)器端模擬器這次是我們第一次使用tomcat和coyote，因此懷疑是tomcat的問(wèn)題，在再三追查代碼無(wú)果的情況下，決定換一個(gè)服務(wù)器端模擬器來(lái)確認(rèn)問(wèn)題所在：到底是cxf的客戶端的問(wèn)題，還是服務(wù)器端模擬器。一個(gè)簡(jiǎn)單的模擬器寫出來(lái)了，一個(gè)跳過(guò)所有業(yè)務(wù)邏輯直接調(diào)用cxf客戶端實(shí)現(xiàn)代碼的測(cè)試小程序?qū)懗鰜?lái)了，測(cè)試之后發(fā)現(xiàn)，問(wèn)題依舊。于是將目光集中到cxf的客戶端上。
在測(cè)試中發(fā)現(xiàn)這樣一個(gè)規(guī)律，在上述服務(wù)器端的異常發(fā)生前，在客戶端中總是會(huì)有規(guī)律的出現(xiàn)下面這個(gè)異常：
Jul 24, 2009 10:36:18 PM org.apache.cxf.phase.PhaseInterceptorChain doIntercept
INFO: Interceptor has thrown exception, unwinding now null
tps = 25
Exception in thread "Thread-41" 2009-07-24 22:36:19,925 149585 [Thread-41] (********Impl.java:459) ERROR junit.framework.Test - Got an exception when invoking **** service:javax.xml.ws.WebServiceException: java.lang.NullPointerException
(這里的信息是和業(yè)務(wù)相關(guān)的，不方便打出，總之和我們討論的問(wèn)題無(wú)關(guān))
at test.TestMci.execute(TestMci.java:84)
at test.TestMci.access$1(TestMci.java:81)
at test.TestMci$TestThread.run(TestMci.java:90)
at java.lang.Thread.run(Thread.java:595)
Caused by: javax.xml.ws.WebServiceException: java.lang.NullPointerException
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:142)
at $Proxy40.authorizeAndPurchase(Unknown Source)
at *********************
6 more
Caused by: java.lang.NullPointerException
at org.apache.cxf.transport.http.HTTPConduit.prepare(HTTPConduit.java:483)
at org.apache.cxf.interceptor.MessageSenderInterceptor.handleMessage(MessageSenderInterceptor.java:46)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:226)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:469)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:299)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:251)
at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:73)
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:124)
8 more
看來(lái)問(wèn)題是出現(xiàn)在這里了。
進(jìn)一步的測(cè)試發(fā)現(xiàn)，低壓力下比如2-3個(gè)工作線程，基本不會(huì)有任何問(wèn)題，因此可以解釋為什么功能測(cè)試時(shí)不出現(xiàn)問(wèn)題。工作線程加到10個(gè)，基本上還算問(wèn)題，只有極其偶然的會(huì)出現(xiàn)一次兩次這個(gè)異常。進(jìn)一步加大工作線程，由于受到測(cè)試機(jī)器的性能限制，10個(gè)工作線程和100個(gè)工作線程的tps基本相同，都大體在300TPS左右(客戶端在筆記本上跑的，cpu已經(jīng)90%+了)。測(cè)試的結(jié)果是上面的異常開(kāi)始變的有規(guī)律，恩，非常搞笑的規(guī)律，大概每10000次左右的請(qǐng)求就發(fā)生一次上述異常，非常的穩(wěn)定而執(zhí)著的重現(xiàn)。服了,這么有規(guī)律而重現(xiàn)性極好的錯(cuò)誤，還真是難得一見(jiàn)........
分析一下問(wèn)題，在tps基本保持不變的情況下，客戶端線程從10增加到100問(wèn)題就變得明顯。因此問(wèn)題的焦點(diǎn)直指線程安全這個(gè)老大難問(wèn)題，重新審視我們使用cxf的代碼，發(fā)現(xiàn)有個(gè)地方
/**
* the server service
*/
MessagingChannelServiceImplService mcsis = null;
/**
* the sever service port
*/
MessagingChannelService mcs = null;
mcsis = new MessagingChannelServiceImplService(serviceWsdlUrl);
mcs = mcsis.getMessagingChannelServiceImplPort();
MessagingChannelService是cxf自動(dòng)生成的，這個(gè)是@WebService，其他的業(yè)務(wù)代碼都是調(diào)用它上面的業(yè)務(wù)方法來(lái)實(shí)現(xiàn)。由于serviceWsdlUrl不變，因此我們重用了一些東西，避免每次都初始化一次。看來(lái)問(wèn)題出現(xiàn)在這里，試著將代碼修改為threadlocal，讓每個(gè)線程都初始化一次然后保存給自己使用。
修改后的客戶端代碼在之后的測(cè)試中，非常穩(wěn)定,沒(méi)有再出現(xiàn)上面的異常，問(wèn)題算是解決了。
我對(duì)cxf不是很熟悉，找了一下也沒(méi)有找到到底是那里造成的線程不安全，google了一下找到幾個(gè)地方，但是似乎還不能完全說(shuō)明問(wèn)題。先列出來(lái)慢慢研究：
1) Are JAX-WS client proxies thread safe?
裝載自這里：
http://cxf.apache.org/faq.html#FAQ-AreJAXWSclientproxiesthreadsafe%253F
Official JAX-WS answer: No. According to the JAX-WS spec, the client proxies are NOT thread safe. To write portable code, you should treat them as non-thread safe and synchronize access or use a pool of instances or similar.
CXF answer: CXF proxies are thread safe for MANY use cases. The exceptions are:
* Use of ((BindingProvider)proxy).getRequestContext() - per JAX-WS spec, the request context is PER INSTANCE. Thus, anything set there will affect requests on other threads. With CXF, you can do:
((BindingProvider)proxy).getRequestContext().put("thread.local.request.context", "true");
((BindingProvider)proxy).getRequestContext().put("thread.local.request.context", "true");
and future calls to getRequestContext() will use a thread local request context. That allows the request context to be threadsafe. (Note: the response context is always thread local in CXF)
* Settings on the conduit - if you use code or configuration to directly manipulate the conduit (like to set TLS settings or similar), those are not thread safe. The conduit is per-instance and thus those settings would be shared.
* Session support - if you turn on sessions support (see jaxws spec), the session cookie is stored in the conduit. Thus, it would fall into the above rules on conduit settings and thus be shared across threads.
For the conduit issues, you COULD install a new ConduitSelector that uses a thread local or similar. That's a bit complex though.
For most "simple" use cases, you can use CXF proxies on multiple threads. The above outlines the workarounds for the others.
2) cxf的wiki中談到Client API中的Proxy-based API
wiki 地址:
http://cwiki.apache.org/CXF20DOC/jax-rs.html
Limitations
Proxy methods can not have @Context method parameters and subresource methods returning Objects can not be invoked - perhaps it is actually not too bad at all - please inject contexts as field or bean properties and have subresource methods returning typed classes : interfaces, abstract classes or concrete implementations.
Proxies are currently not thread-safe.
3) thread safe issue caused by XMLOutputFactoryImpl
找到的一個(gè)cxf的bug，
https://issues.apache.org/jira/browse/CXF-2229
Description
Currently CXF calls StaxUtils.getXMLOutputFactory() to get the cached instance of XMLOutputFactoryImpl. But XMLOutputFactoryImpl.createXMLStreamWriter is not thread-safe. See below.
javax.xml.stream.XMLStreamWriter createXMLStreamWriter(javax.xml.transform.stream.StreamResult sr, String encoding) throws javax.xml.stream.XMLStreamException {
try{
if(fReuseInstance && fStreamWriter != null && fStreamWriter.canReuse() && !fPropertyChanged){
fStreamWriter.reset();
fStreamWriter.setOutput(sr, encoding);
if(DEBUG)System.out.println("reusing instance, object id : " + fStreamWriter);
return fStreamWriter;
}
return fStreamWriter = new XMLStreamWriterImpl(sr, encoding, new PropertyManager(fPropertyManager)); -- this is not thread safe, since the new instance is assigned to the field fStreamWriter first, then it is possible that different threads get the same XMLStreamWriterImpl when they call this method at the same time.
}catch(java.io.IOException io){
throw new XMLStreamException(io);
}
}
The solution might be, StaxUtils.getXMLOutputFactory() method creates a new instance of XMLOutputFactory every time, don't cache it.