Monday, November 17, 2014
But that's impossible, or finding out that the JIT has broken your code.
Every now and then you look at some code and think that it cannot be possibly be wrong. Once you have ruled out a simple programmer screw up / enemy action in code (Make sure you read Java Puzzlers or similar) or a concurrency issue (Read Java Concurrency or go on Dr Heniz excellent course) you should sit back and take a few days and then starting thinking about whether the JDK is indeed out to get you. I haven't seen one in the wild in my 18 odd years as a Java programmer so it kinda took me by surprise.
If you are running against JDK 8 in a large scale Swing application you might eventually see the following exception, lots of lots of times. (Unless you forgot the lesson learned in my previous blog in your logging code, in which case you might see lots of ArrayOfOutBoundsException)
This error is particularly insidious because it takes around ten minutes to show itself and sometimes not at all. If you look at the code for this class the line in question, starts with "startsFrom = break", only accesses two local variables both of which have been previously referenced in the method.
The most direct way to rule out a JIT error is to disable compilation for just this single method, here is an example; but you can fine more in the documentation for the command line java tool.
When this parameter is added, the problem goes away. - since we have ruled out enemy action by code or a concurrency issue we can be more sure this is a JIT issue. Now as part of the bug logging for this I output diagnostics for this single method, and found out that the problem, didn't occur until the method was JITted for a fifth time.
Here is some diagnostic output seen with the above command:
Now I am still working with the JDK team for a fix for this one; but I do feel I have discovered a useful set of tools for providing some evidence that the JIT compiler is causing my bad day. And more importantly I have a workaround so I can run my tests until this is resolved.
If you are running against JDK 8 in a large scale Swing application you might eventually see the following exception, lots of lots of times. (Unless you forgot the lesson learned in my previous blog in your logging code, in which case you might see lots of ArrayOfOutBoundsException)
Caused by: java.lang.NullPointerException at javax.swing.text.GlyphView.getBreakSpot(GlyphView.java:799) at javax.swing.text.GlyphView.getBreakWeight(GlyphView.java:724) at javax.swing.text.FlowView$LogicalView.getPreferredSpan(FlowView.java:733) at javax.swing.text.FlowView.calculateMinorAxisRequirements(FlowView.java:233) at javax.swing.text.ParagraphView.calculateMinorAxisRequirements(ParagraphView.java:717) at javax.swing.text.BoxView.checkRequests(BoxView.java:935) at javax.swing.text.BoxView.getMinimumSpan(BoxView.java:568) at javax.swing.text.BoxView.calculateMinorAxisRequirements(BoxView.java:903) at javax.swing.text.BoxView.checkRequests(BoxView.java:935) at javax.swing.text.BoxView.setSpanOnAxis(BoxView.java:343) at javax.swing.text.BoxView.layout(BoxView.java:708) at javax.swing.text.BoxView.setSize(BoxView.java:397) ...
This error is particularly insidious because it takes around ten minutes to show itself and sometimes not at all. If you look at the code for this class the line in question, starts with "startsFrom = break", only accesses two local variables both of which have been previously referenced in the method.
Segment s = getText(pstart, pend);
s.first();
BreakIterator breaker = getBreaker();
breaker.setText(s);
// Backward search should start from end+1 unless there's NO end+1
int startFrom = end + (pend > end ? 1 : 0);
for (;;) {
startFrom = breaker.preceding(s.offset + (startFrom - pstart))
+ (pstart - s.offset);
if (startFrom > start) {
// The break spot is within the view
bs[ix++] = startFrom;
} else {
break;
}
}
The most direct way to rule out a JIT error is to disable compilation for just this single method, here is an example; but you can fine more in the documentation for the command line java tool.
javaThing -XX:CompileCommand=exclude,javax/swing/text/GlyphView,getBreakSpot
When this parameter is added, the problem goes away. - since we have ruled out enemy action by code or a concurrency issue we can be more sure this is a JIT issue. Now as part of the bug logging for this I output diagnostics for this single method, and found out that the problem, didn't occur until the method was JITted for a fifth time.
javaThing -XX:CompileCommand=print,javax/swing/text/GlyphView,getBreakSpot
Here is some diagnostic output seen with the above command:
Compiled method (c2) 914078 33142 4 javax.swing.text.GlyphView::getBreakSpot (247 bytes)
total in heap [0x00002aaab0749e10,0x00002aaab0750fe0] = 29136
relocation [0x00002aaab0749f38,0x00002aaab074a1e8] = 688
constants [0x00002aaab074a200,0x00002aaab074a2a0] = 160
main code [0x00002aaab074a2a0,0x00002aaab074cde0] = 11072
stub code [0x00002aaab074cde0,0x00002aaab074ce40] = 96
oops [0x00002aaab074ce40,0x00002aaab074ce58] = 24
metadata [0x00002aaab074ce58,0x00002aaab074d058] = 512
scopes data [0x00002aaab074d058,0x00002aaab074ea20] = 6600
scopes pcs [0x00002aaab074ea20,0x00002aaab0750c50] = 8752
dependencies [0x00002aaab0750c50,0x00002aaab0750c80] = 48
handler table [0x00002aaab0750c80,0x00002aaab0750e90] = 528
nul chk table [0x00002aaab0750e90,0x00002aaab0750fe0] = 336
OopMapSet contains 113 OopMaps
#0
OopMap{[8]=Oop [32]=Oop [40]=Oop off=892}
#1
OopMap{[32]=Oop [40]=Oop off=960}
#2
OopMap{[32]=Oop [40]=Oop off=980}
#3
OopMap{[32]=Oop [40]=Oop [48]=Oop off=1048}
#4
OopMap{[32]=Oop [40]=Oop [48]=Oop off=1084}
#5
OopMap{[0]=Oop [24]=Oop [48]=Oop [56]=Oop [80]=Oop off=2500}
#6
OopMap{rbx=Oop rdi=Oop [32]=Oop [40]=Oop [112]=Oop off=2533}
#7
OopMap{rbx=Oop rdi=Oop r14=Oop [32]=Oop [112]=Oop off=3081}
#8
OopMap{rbx=Oop rdi=Oop r14=Oop [32]=Oop [40]=Oop [112]=Oop off=3190}
#9
OopMap{[8]=Oop [32]=Oop [40]=Oop off=4408}
#10
OopMap{[32]=Oop [40]=Oop [48]=Oop off=4640}
#11
OopMap{rbp=Oop [16]=Oop [24]=Oop [40]=Oop [64]=Oop off=5232}
#12
OopMap{rbp=Oop [0]=NarrowOop [32]=Oop off=5364}
#13
OopMap{[32]=Oop [40]=Oop [48]=Oop off=5408}
#14
OopMap{rbp=Oop [32]=Oop [40]=Oop [48]=Oop off=5436}
#15
OopMap{rbp=Oop [32]=Oop [40]=Oop [48]=Oop off=5468}
#16
OopMap{rbp=Oop [32]=Oop [40]=Oop [48]=Oop off=5524}
#17
OopMap{rbp=Oop [32]=Oop [40]=Oop [48]=Oop [88]=Oop off=5552}
#18
OopMap{[32]=Oop [40]=Oop [48]=Oop [64]=Oop [72]=Derived_oop_[64] [112]=Oop off=5608}
#19
OopMap{[8]=Oop [32]=Oop off=5680}
#20
OopMap{rbp=Oop off=5720}
#21
OopMap{rbp=Oop off=5752}
#22
OopMap{rbp=Oop [24]=NarrowOop [28]=NarrowOop [32]=Oop [40]=Oop [48]=Oop [56]=Oop [64]=Oop [88]=Oop off=5812}
#23
OopMap{rbp=Oop [16]=Oop [24]=Oop [40]=Oop [64]=Oop [88]=Oop off=5960}
#24
OopMap{[0]=Oop [24]=Oop [48]=Oop [56]=Oop [72]=Oop [88]=NarrowOop off=6056}
#25
OopMap{[40]=Oop off=6088}
#26
OopMap{[0]=Oop off=6120}
#27
OopMap{[8]=Oop [24]=Oop [56]=Oop [72]=Oop [112]=Oop off=6216}
#28
OopMap{[0]=Oop [32]=NarrowOop [40]=Oop off=6284}
#29
OopMap{rbp=Oop [16]=Oop [40]=Oop [64]=Oop [112]=Oop off=6384}
#30
OopMap{[0]=Oop off=6412}
#31
OopMap{[0]=Oop [16]=Oop [32]=NarrowOop [40]=Oop [48]=Oop off=6488}
#32
OopMap{rbp=Oop [16]=Oop [40]=Oop [48]=Oop off=6560}
#33
OopMap{[32]=Oop [40]=Oop [48]=Oop [64]=Oop [112]=Oop off=6608}
#34
OopMap{[8]=Oop [28]=NarrowOop [32]=Oop [40]=Oop [48]=Oop off=6768}
#35
OopMap{rbp=NarrowOop [0]=Oop [16]=Oop [32]=Oop [40]=NarrowOop off=6860}
#36
OopMap{[0]=Oop [16]=Oop [32]=NarrowOop [40]=Oop [48]=Oop off=6988}
#37
OopMap{rbp=Oop [32]=Oop off=7024}
#38
OopMap{rbp=NarrowOop [0]=Oop [24]=Oop [32]=Oop off=7260}
#39
OopMap{rbp=NarrowOop [0]=Oop [24]=Oop [32]=Oop off=7344}
#40
OopMap{rbp=Oop [16]=Oop [24]=Oop [40]=Oop [60]=NarrowOop [64]=Oop off=7452}
#41
OopMap{rbp=NarrowOop [32]=Oop off=7476}
#42
OopMap{rbp=NarrowOop [0]=Oop off=7524}
#43
OopMap{[32]=Oop [40]=Oop [48]=Oop off=7588}
#44
OopMap{[32]=Oop [40]=Oop [48]=Oop off=7616}
#45
OopMap{[32]=Oop [40]=Oop [48]=Oop off=7632}
#46
OopMap{rbp=NarrowOop [32]=Oop off=7676}
#47
OopMap{rbp=NarrowOop [0]=Oop off=7724}
#48
OopMap{[0]=Oop [16]=Oop [28]=NarrowOop [40]=Oop [48]=Oop [56]=NarrowOop [64]=Oop off=7868}
#49
OopMap{[8]=Oop [28]=NarrowOop [32]=Oop [40]=Oop [48]=Oop [56]=Oop off=7916}
#50
OopMap{rbp=Oop [16]=Oop [24]=NarrowOop off=8016}
#51
OopMap{rbp=Oop [16]=Oop [28]=NarrowOop off=8080}
#52
OopMap{rbp=NarrowOop [0]=Oop [24]=Oop [32]=Oop off=8152}
#53
OopMap{rbp=Oop [8]=NarrowOop off=8212}
#54
OopMap{rbp=NarrowOop [32]=Oop off=8236}
#55
OopMap{rbp=Oop [16]=NarrowOop off=8272}
#56
OopMap{rbp=NarrowOop [0]=Oop off=8320}
#57
OopMap{rbp=Oop [12]=NarrowOop off=8360}
#58
OopMap{rbp=NarrowOop [32]=Oop off=8400}
#59
OopMap{rbp=Oop [12]=NarrowOop off=8460}
#60
OopMap{rbp=NarrowOop [0]=Oop off=8508}
#61
OopMap{rbp=Oop [24]=NarrowOop [40]=Oop off=8572}
#62
OopMap{rbp=Oop off=8600}
#63
OopMap{rbp=Oop [8]=Oop [28]=NarrowOop off=8640}
#64
OopMap{rbp=Oop [8]=Oop [20]=NarrowOop [112]=Oop off=8704}
#65
OopMap{rbp=Oop [16]=Oop [24]=Oop [48]=Oop off=8788}
#66
OopMap{rbp=Oop [16]=Oop [24]=Oop [40]=Oop [64]=Oop off=8912}
#67
OopMap{rbp=Oop [16]=Oop [24]=Oop [40]=Oop [64]=Oop off=9036}
#68
OopMap{rbp=Oop [16]=Oop [24]=Oop [40]=Oop [64]=Oop off=9160}
#69
OopMap{rbp=Oop [16]=Oop [24]=Oop [40]=Oop [64]=Oop off=9284}
#70
OopMap{rbp=Oop [16]=Oop [24]=Oop [40]=Oop [64]=Oop off=9408}
#71
OopMap{rbp=Oop [16]=Oop [24]=Oop [40]=Oop [64]=Oop off=9532}
#72
OopMap{off=9556}
#73
OopMap{off=9580}
#74
OopMap{off=9604}
#75
OopMap{[112]=Oop off=9628}
#76
OopMap{rbp=Oop [8]=Oop [24]=Oop [32]=NarrowOop off=9696}
#77
OopMap{rbp=Oop [8]=Oop [24]=NarrowOop off=9760}
#78
OopMap{off=9784}
#79
OopMap{off=9812}
#80
OopMap{off=9836}
#81
OopMap{off=9860}
#82
OopMap{off=9884}
#83
OopMap{off=9908}
#84
OopMap{off=9932}
#85
OopMap{off=9956}
#86
OopMap{off=9980}
#87
OopMap{off=10004}
#88
OopMap{off=10028}
#89
OopMap{rbp=Oop [16]=Oop [28]=NarrowOop off=10092}
#90
OopMap{rbp=Oop [16]=Oop [24]=Oop [48]=Oop off=10176}
#91
OopMap{off=10200}
#92
OopMap{off=10224}
#93
OopMap{off=10248}
#94
OopMap{off=10272}
#95
OopMap{off=10296}
#96
OopMap{off=10320}
#97
OopMap{off=10344}
#98
OopMap{off=10368}
#99
OopMap{off=10392}
#100
OopMap{off=10416}
#101
OopMap{off=10440}
#102
OopMap{off=10464}
#103
OopMap{off=10488}
#104
OopMap{off=10512}
#105
OopMap{off=10536}
#106
OopMap{off=10560}
#107
OopMap{off=10584}
#108
OopMap{off=10608}
#109
OopMap{off=10632}
#110
OopMap{off=10656}
#111
OopMap{off=10680}
#112
OopMap{off=11028}
java.lang.NullPointerException
at javax.swing.text.GlyphView.getBreakSpot(GlyphView.java:799)
at javax.swing.text.GlyphView.getBreakWeight(GlyphView.java:724)
at javax.swing.text.html.InlineView.getBreakWeight(InlineView.java:150)
at javax.swing.text.FlowView$LogicalView.getPreferredSpan(FlowView.java:733)
at javax.swing.text.FlowView.calculateMinorAxisRequirements(FlowView.java:233)
at javax.swing.text.ParagraphView.calculateMinorAxisRequirements(ParagraphView.java:717)
at javax.swing.text.html.ParagraphView.calculateMinorAxisRequirements(ParagraphView.java:157)
at javax.swing.text.BoxView.checkRequests(BoxView.java:935)
at javax.swing.text.BoxView.getMinimumSpan(BoxView.java:568)
at javax.swing.text.html.ParagraphView.getMinimumSpan(ParagraphView.java:270)
at javax.swing.text.BoxView.calculateMinorAxisRequirements(BoxView.java:903)
Now I am still working with the JDK team for a fix for this one; but I do feel I have discovered a useful set of tools for providing some evidence that the JIT compiler is causing my bad day. And more importantly I have a workaround so I can run my tests until this is resolved.
Labels:
java
Thursday, November 6, 2014
Missing stack traces for repeated exceptions (reminder)
A long while ago a optimisation was added to the JVM so that if the same exception is thrown again and again and again a single instance of the Exception is created without the stack trace filled in in order to increase performance. This is an excellent idea unless you are trying to diagnose a problem and you have missed the original error.
If you forgot about this optimisation you send the afternoon looking at the following log output and weeping slightly. (In my defence I have a little one in the house hence the fuzzy brain and lack of blogging action this year)
It turns out that you can turn off this optimisation with a simple flag:
In my particular case the actual exception causing this trouble was a NPE from the GlyphView code in JDK8. (One that is being caused by a glitch in hotspot it seems) But that in turn was causing the AIOOBE in some logging code clouding the issue even more.
This in particular is a good flag to add by default when running your automated tests, particularly in combination with the stack trace length override I have talked about before.
If you forgot about this optimisation you send the afternoon looking at the following log output and weeping slightly. (In my defence I have a little one in the house hence the fuzzy brain and lack of blogging action this year)
java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException ... java.lang.Exception: Uncaught exceptions during test at oracle.jdevstudio-testware-tests.level0.testADFcMATS(/.../work/mw9111/jdeveloper/jdev/extensions/oracle.jdevstudio-testware-tests/abbot/common-adf/CreateNewCustomAppAndProjectWithName.xml:25) at oracle.jdevstudio-testware-tests.level0.testADFcMATS(/.../work/mw9111/jdeveloper/jdev/extensions/oracle.jdevstudio-testware-tests/abbot/level0/testADFcMATS.xml:94) at oracle.abbot.JDevScriptFixture.runTest(JDevScriptFixture.java:555) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:243) at junit.framework.TestSuite.run(TestSuite.java:238) at junit.textui.TestRunner.doRun(TestRunner.java:116) at junit.textui.TestRunner.doRun(TestRunner.java:109) at oracle.abbot.AbbotRunner.run(AbbotRunner.java:614) at oracle.abbot.AbbotAddin$IdeAbbotRunner.run(AbbotAddin.java:634) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.ArrayIndexOutOfBoundsException [Crickets]
It turns out that you can turn off this optimisation with a simple flag:
java -XX:-OmitStackTraceInFastThrow ....
In my particular case the actual exception causing this trouble was a NPE from the GlyphView code in JDK8. (One that is being caused by a glitch in hotspot it seems) But that in turn was causing the AIOOBE in some logging code clouding the issue even more.
This in particular is a good flag to add by default when running your automated tests, particularly in combination with the stack trace length override I have talked about before.
Labels:
java
Subscribe to:
Posts (Atom)
