The document discusses various myths and misconceptions in software engineering. It summarizes research that analyzed large codebases to determine factors correlated with defects. Metrics like lines of code and complexity metrics correlated with defects, but the relationship depended on the project. Defect-prone modules could be predicted by combining metrics. Other factors like developer experience, prior defects, language features, and dependencies also influenced bugs. While some properties were project-specific, others like the role of requirements, design, coding and testing were universal defect sources.

1. Myths in Software Engineering
“Bugs Reside in Complex Code”
Andreas Zeller
Saarland University

4. Obtaining Data

5. Models Specs Code Traces Profiles Tests
e-mail Bugs Effort Navigation Changes Chats

6. Models Specs Code Traces Profiles Tests
e-mail Bugs Effort Navigation Changes Chats

7. Bugs Changes

8. Bugs Changes

9. Bugs Changes

10. Bugs Changes

11. Bugs Changes

12. Bugs Changes

13. Map bugs to
code locations
Bugs Changes

14. Eclipse Bugs

15. What is the cause
Eclipse Bugs
of these errors?

16. Code complexity
Past
defects

17. Code complexity
Past
defects
#Lines #Vars #Classes
Metrics
#Params #Reads #Writes
#Arcs #Blocks McCabe
Fan In Fan Out …

18. Projects researched
• Internet Explorer 6
• IIS Server
• Windows Process Messaging
• DirectX
• NetMeeting
>1,000,000 Lines of Code

19. Projects researched
>1,000,000 Lines of Code

20. Projects researched
ABCDE
>1,000,000 Lines of Code

21. Do metrics correlate
with defect density?

22. Do metrics correlate
with defect density?
Project Metrics correlated w/ defects
A #Classes and 5 derived
B almost all
C all except MaxInheritanceDepth
D only #Lines
E #Functions, #Arcs, Complexity

23. Do metrics correlate
with defect density?
Project Metrics correlated w/ defects
A #Classes and 5 derived
B almost all
C all except MaxInheritanceDepth
D only #Lines
E #Functions, #Arcs, Complexity

24. Do metrics correlate
with defect density?
Project Metrics correlated w/ defects
A #Classes and 5 derived
B almost all
C all except MaxInheritanceDepth
D only #Lines
E #Functions, #Arcs, Complexity

25. Do metrics correlate
with defect density?
Project Metrics correlated w/ defects
A #Classes and 5 derived
B almost all
C all except MaxInheritanceDepth
D only #Lines
E #Functions, #Arcs, Complexity

26. Do metrics correlate
with defect density?
Project Metrics correlated w/ defects
A #Classes and 5 derived
B almost all
C all except MaxInheritanceDepth
D only #Lines
E #Functions, #Arcs, Complexity

27. Do metrics correlate
with defect density?
Project Metrics correlated w/ defects
A #Classes and 5 derived
B almost all
C all except MaxInheritanceDepth
D only #Lines
E #Functions, #Arcs, Complexity

28. Do metrics correlate
with defect density?
YES
Project Metrics correlated w/ defects
A #Classes and 5 derived
B almost all
C all except MaxInheritanceDepth
D only #Lines
E #Functions, #Arcs, Complexity

29. Is there a set of metrics
that fits all projects?
Project Metrics correlated w/ defects
A #Classes and 5 derived
B almost all
C all except MaxInheritanceDepth
D only #Lines
E #Functions, #Arcs, Complexity

30. Is there a set of metrics
that fits all projects?
NO
Project Metrics correlated w/ defects
A #Classes and 5 derived
B almost all
C all except MaxInheritanceDepth
D only #Lines
E #Functions, #Arcs, Complexity

31. Can we predict
defect-prone modules?

32. Can we predict
defect-prone modules?
• Basic idea: Combine metrics

33. Can we predict
defect-prone modules?
• Basic idea: Combine metrics
• Give most weight to most predictive metrics

34. Can we predict
defect-prone modules?
• Basic idea: Combine metrics
• Give most weight to most predictive metrics
• Successful prediction in all five projects –

35. Can we predict
defect-prone modules?
• Basic idea: Combine metrics
• Give most weight to most predictive metrics
• Successful prediction in all five projects –
• – but requires history to calibrate

36. Assistance

37. Eclipse Bugs

38. What is the cause
Eclipse Bugs
of these errors?

39. Is it the developers?

40. Is it the developers?
Does experience
matter?

41. Is it the developers?
Bug density
Does experience
correlates with
matter?
experience!

42. Is it history?

43. Is it history?
I found lots of
bugs here. Will
there be more?

44. Is it history?
I found lots of Yes! (But where
bugs here. Will did these come
there be more? from?)

45. How about metrics?

46. How about metrics?
Do code metrics
correlate with bug
density?

47. How about metrics?
Do code metrics
Sometimes!
correlate with bug
density?

48. Uh. Coverage?

49. Uh. Coverage?
Does test coverage
correlate with bug
density?

50. Uh. Coverage?
Yes –
Does test coverage
correlate with bug the more coverage,
density? the more bugs!

51. Ah! Language features?

52. Ah! Language features?
Are gotos
harmful?

53. Ah! Language features?
Are gotos
No correlation!
harmful?

54. Ok. Problem domain?

55. Ok. Problem domain?
Which tokens
do matter?

56. Ok. Problem domain?
Which tokens import • extends
• implements
do matter?

57. Eclipse imports
Joint work with Adrian Schröter • Tom Zimmermann

58. Eclipse imports
import org.eclipse.jdt.internal.compiler.lookup.*;
import org.eclipse.jdt.internal.compiler.*;
import org.eclipse.jdt.internal.compiler.ast.*;
import org.eclipse.jdt.internal.compiler.util.*;
...
import org.eclipse.pde.core.*;
import org.eclipse.jface.wizard.*;
import org.eclipse.ui.*;
Joint work with Adrian Schröter • Tom Zimmermann

59. Eclipse imports
71% of all components importing compiler
show a post-release defect
import org.eclipse.jdt.internal.compiler.lookup.*;
import org.eclipse.jdt.internal.compiler.*;
import org.eclipse.jdt.internal.compiler.ast.*;
import org.eclipse.jdt.internal.compiler.util.*;
...
import org.eclipse.pde.core.*;
import org.eclipse.jface.wizard.*;
import org.eclipse.ui.*;
Joint work with Adrian Schröter • Tom Zimmermann

60. Eclipse imports
71% of all components importing compiler
show a post-release defect
import org.eclipse.jdt.internal.compiler.lookup.*;
import org.eclipse.jdt.internal.compiler.*;
import org.eclipse.jdt.internal.compiler.ast.*;
import org.eclipse.jdt.internal.compiler.util.*;
...
import org.eclipse.pde.core.*;
import org.eclipse.jface.wizard.*;
import org.eclipse.ui.*;
14% of all components importing ui
show a post-release defect
Joint work with Adrian Schröter • Tom Zimmermann

61. Mozilla Vulnerabilities

62. And what else?

63. And what else?
So all of this
requires earlier
defects, right?

64. And what else?
So all of this
requires earlier Yes!
defects, right?

65. And what else?

66. And what else?
But are there
universal
properties?

67. And what else?
?
But are there
universal
properties?

68. Defect
sources

69. Defect
sources
Coding

70. Defect
sources
Coding Quality Assurance

71. Defect
sources
Design
Coding Quality Assurance

72. Defect
sources
Design
Requirements
Coding Quality Assurance

73. Defect
sources
• Which properties Design
Requirements
should we look at?
• Which properties
can we look at?
Coding Quality Assurance

74. bug density
Plugin.java had 5 failures )
before and one failure after
release (``post''). The
package contains 43 files
(``points'') and encountered 16
failures before and one failure
after release; on average each
file in this package had 0.609
failures before and 0.022
failures after release (``avg'')
Bugs • Fixes • Changes

75. bug density
Plugin.java had 5 failures )
before and one failure after
release (``post''). The
package contains 43 files
(``points'') and encountered 16
failures before and one failure
after release; on average each
file in this package had 0.609
failures before and 0.022
failures after release (``avg'')
Bugs • Fixes • Changes

76. <?xml version=quot;1.0quot; encoding=quot;UTF-8quot;?>
<defects project=quot;eclipsequot; release=quot;3.0quot;>
<package name=quot;org.eclipse.core.runtimequot;>
<counts>
<count id=quot;prequot; value=quot;16quot; avg=quot;0.609quot; points=quot;43quot; max=quot;5quot;>
<count id=quot;postquot; value=quot;1quot; avg=quot;0.022quot; points=quot;43quot; max=quot;1quot;>
</counts>
<compilationunit name=quot;Plugin.javaquot;>
<counts>
<count id=quot;prequot; value=quot;5quot;> Plugin.java had 5 failures )
before and one failure after
<count id=quot;postquot; value=quot;1quot;> release (``post''). The
package contains 43 files
(``points'') and encountered 16
failures before and one failure
after release; on average each
file in this package had 0.609
failures before and 0.022
failures after release (``avg'')
Bugs • Fixes • Changes

77. Defect
sources
• Which properties Design
Requirements
should we look at?
• Which properties
can we look at?
Coding Quality Assurance

78. Defect
sources
• Which properties Design
Requirements
should we look at?
• Which properties
can we look at?
Coding Quality Assurance

79. Defect
sources
• Which properties Design
Requirements
should we look at?
• Which properties
can we look at?
Coding Quality Assurance