Andrux&me

Thursday, June 5, 2014

Various conditions and I/O Performance

For previos studies on I/O performances, refer to [1],[2] and [3]

My current set-up :

It slightly changed ...

Motorola MotoG
memory chipset : emmc: 8GB Sandisk REV=06 PRV=07 TYPE=17
CM11-20140603-Nightly
RenderKernel r23
192-1190MHz
I/O scheduler : fiops
gov : is part of the tested conditions!
sdcard read-ahead : is part of the tested conditions!

Benchmark used : AndroBench v3.4

What is being tested?

Sequential Read (MB/s)
Sequential Write (MB/s)
Random Read (IOPS)
Random Write (IOPS)
SQlite insert (TPS)
SQlite update (TPS)
SQlite delete (TPS)

Regarding the following modifications :

CPU governor (Interactive, intelliactive, intellidemand and ondemand), with fixed sdcard read ahead : 128kB
sdcard read ahead (128, 256,512,1024 and 2048kB), with fixed CPU governor (Interactive)

Results :

Changing CPU Governor :

governor	seq read (MB/s)	seq write (MB/s)	rd read (IOPS)	rd write (IOPS)	SQlite insert (TPS)	SQlite update (TPS)	SQlite delete (TPS)
interactive	65,28	15,84	1849,93	393,33	51,58	57,72	58,85
stdev	0,77	0,46	11,03	12,96	1,47	0,93	0,58
intellidemand	54,81	13,45	1117,70	219,32	31,73	34,04	34,22
stdev	2,47	0,66	60,30	11,87	1,02	0,37	0,61
intelliactive	67,56	15,77	1984,63	386,10	53,27	56,90	59,65
stdev	1,36	0,23	64,20	22,67	1,87	1,56	0,36
ondemand	65,47	16,06	1777,45	278,06	46,93	49,73	53,84
stdev	4,43	2,36	63,01	42,24	3,11	3,56	1,20

The following graph shows a delta% against median (100%), that way a little difference looks big (mostly because of the chosen scale)

Changing sdcard read ahead :

sd read ahead (kB)	seq read (MB/s)	seq write (MB/s)	rd read (IOPS)	rd write (IOPS)	SQlite insert (TPS)	SQlite update (TPS)	SQlite delete (TPS)
128	65,28	15,84	1849,93	393,33	51,58	57,72	58,85
stdev	0,77	0,46	11,03	12,96	1,47	0,93	0,58
256	66,37	15,93	1844,57	397,16	51,53	56,78	59,33
stdev	2,61	2,09	17,24	28,20	1,06	1,01	0,13
512	67,04	17,22	1828,12	385,42	51,52	57,14	57,81
stdev	2,19	3,94	30,43	21,74	1,29	0,80	0,34
1024	65,90	18,50	1999,54	384,88	51,37	55,92	58,42
stdev	5,59	1,04	25,79	3,23	1,54	2,43	0,46
2048	69,01	16,47	1962,58	383,69	51,32	56,91	58,84
stdev	4,82	1,10	20,62	4,66	0,72	1,69	1,22

The following graph shows a delta% against median (100%), that way a little difference looks big (mostly because of the chosen scale)

What does that show?

Regarding governors we can say interactive and intelliactive are pretty similar and that intellidemand is far behind. (stdev are not really good, but differences between averages are higher than standard deviations)
Regarding cache size (sdcard read ahead), results are totally inconsistent (look at stdev/averages!), a 'No Conclusion' conclusion is safe since differences don't look significant.
I remember that in the X10mini ages (2010), any size higher than 128kB created hiccups during music playback ... Worth the risk?

==> What we can say is that I/O performances aren't only linked to IO-schedulers but to many other critical points such as CPU governor, (maybe) sdcard cache size, and other untested conditions.

What are the limits?
same as before ...

Lack of accuracy with only 3 test per condition (and high results dispersion)

These results are at least device dependent, kernel dependent, ROM dependent AND emmc type dependent (MotoG could be built with at least 4 different memory chips)
CPU governors are highly tunable, so one that is bad here could be better in other conditions ...

Delta% against median is not the best to compare but was the easiest here (i could not put IOPS and MB/s in the same chart because of scaling)

Don't forget to read other I/O performances related posts :

I/O schedulers and performance

I/O schedulers and performance -2-

I/O schedulers and performance -3-

Wednesday, May 21, 2014

I/O schedulers and performance -3-

I/O schedulers and their impact on performance

(part 3)

for setup refer to part 2

What is being tested ?

SQlite insert
SQlite update
SQlite delete

==> Using AndroBench v3.4 default settings

==> Each available scheduler is tested 3 times (same schedulers as previous post, i tried 10 times with fiops, but results are still inaccurate, so 3 times is enough not to waste time.)

Results :
RAW data : the table displays mean result (out of 3) and related standard deviation to show how (in)consistent some results are.

in red : standard deviation that overrides the cut off (>2)

all results in TPS (Transaction per second)

IO scheduler	SQlite insert	SQlite update	SQlite delete	total
fiops	40,97	41,07	41,48	41,17
stdev	1,49	1,59	2,09	1,72
noop	42,21	40,13	42,08	41,47
stdev	0,91	2,60	1,76	1,76
bfq	41,35	35,26	36,94	37,85
stdev	1,70	2,32	1,98	2,00
cfq	40,82	40,70	40,87	40,79
stdev	2,96	1,54	1,03	1,84
deadline	40,84	38,75	41,86	40,48
stdev	2,62	0,86	3,60	2,36
zen	42,93	38,90	40,11	40,65
stdev	0,36	1,23	2,11	1,23
vr	41,79	38,67	41,00	40,49
stdev	0,67	1,74	1,07	1,16
row	39,09	41,32	40,75	40,39
stdev	5,07	2,65	3,18	3,63
sio	39,88	41,87	42,16	41,30
stdev	2,28	2,23	1,56	2,02

Following graphs display Sequential and Random Read/Write per Scheduler

As we can see, results are not much different from a scheduler to another

Though we can see that bfq is a bit behind

To better discriminate the good and the evil, i tried to take the median result as 100% (median is not modified by extreme values contrary to average/mean)

==> remember : this is a delta% against median (100%), that way a little difference looks big (mostly because of the chosen scale)

How to read the graph?

Same as previous post

What does that show?

A lot, and it is not easy to extract things from it :
A scheduler that is good in SQlite insert is bat at update (and vice versa) : zen and sio mostly
fiops is balanced but not extraordinary performant

Let's see the global graph

fiops seems to stay at the first place (mostly thanks to sequential read and write, it is not the best for SQlite)
noop is the more balance and can be a good alternative
bfq is clearly the worst here.

Since i'm not really sure how to conclude .... do it yourself ;)

==> These results are not god's speaking, and may be different with other tests/benchmarks AND devices.

What are the limits?

Lack of accuracy with only 3 test per scheduler (and high results dispersion)
These results are device dependent, kernel dependent, ROM dependent AND emmc type dependent (MotoG could be built with at least 4 different memory chips)
Delta% against median is not the best to compare but was the easiest here (i could not put TPS, IOPS and MB/s in the same chart because of scaling)

Don't forget to read other I/O performances related posts :

I/O schedulers and performance

I/O schedulers and performance -2-

Various conditions and I/O Performance

Tuesday, May 20, 2014

I/O schedulers and performance -2-

I/O schedulers and their impact on performance

(part 2)

For details about I/O schedulers (what are they?) refer to the previous article

My current set up :

Motorola MotoG
memory chipset : emmc: 8GB Sandisk REV=06 PRV=07 TYPE=17
CM11-20140517-Nightly
RenderKernel r17
192-1190MHz
gov : intellidemand

Benchmark used : AndroBench v3.4

What is being tested ?

Sequential Read (MB/s)
Sequential Write (MB/s)
Random Read (IOPS)
Random Write (IOPS)

==> Using AndroBench v3.4 default settings

==> Each available scheduler is tested 3 times

fiops
noop
bfq
cfq
deadline
zen
vr
row
sio

Results :
RAW data : the table displays mean result (out of 3) and related standard deviation to show how (in)consistent some results are.

IO scheduler	seq read (MB/s)	seq write (MB/s)	rd read (IOPS)	rd write (IOPS)
fiops	62,76	15,61	1379,42	236,48
stdev	3,39	2,77	21,64	17,15
noop	59,34	13,43	1329,92	231,53
stdev	3,51	0,42	44,79	4,30
bfq	54,62	12,12	1320,23	231,39
stdev	1,53	0,49	25,45	2,11
cfq	55,31	14,37	1322,95	226,21
stdev	0,59	0,98	1,22	14,56
deadline	53,37	13,18	1335,51	229,98
stdev	5,51	1,14	26,91	6,46
zen	57,54	13,28	1383,16	232,40
stdev	1,18	1,93	39,88	14,18
vr	53,91	14,54	1352,52	226,86
stdev	1,22	1,15	22,62	1,95
row	57,88	13,66	1351,85	220,26
stdev	0,53	0,90	38,84	14,24
sio	58,32	13,26	1375,62	229,29
stdev	1,27	0,49	18,06	8,30

Following graphs display Sequential and Random Read/Write per Scheduler

The higher the better.

As we can see, results are not much different from a scheduler to another (unsignificant differences for Random IO)

Though we can see that fiops is a bit above the others (Zen, sio, noop and row are not that bad!)

To better discriminate the good and the evil, i tried to take the median result as 100% (median is not modified by extreme values contrary to average/mean)

==> remember : this is a delta% against median (100%), that way a little difference looks big (mostly because of the chosen scale)

How to read the graph?

With median value as 100%,

fiops is 109% in sequential read, 116% in sequential write, 102% in radom read and 103% in radom write
bfq is 94% in sequential read, 90% in sequential write, 97% in radom read and 100% in radom write

What does that show?

fiops seems to be the best here
Zen, noop, sio and row are quite similar
cfq and vr are behind in sequential read but better in sequential write
deadline and bfq are worst here

==> These results are not god's speaking, and may be differents with other tests/benchmarks AND devices.

(i may test SQlite insert, update and delete later, perhaps these will change results?? )

What are the limits?

Lack of accuracy with only 3 test per scheduler (and high results dispersion)
These results are device dependent, kernel dependent, ROM dependent AND emmc type dependent (MotoG could be built with at least 4 different memory chips)
Delta% against median is not the best to compare but was the easiest here (i could not put IOPS and MB/s in the same chart because of scaling)

Don't forget to read other I/O performances related posts :

I/O schedulers and performance

I/O schedulers and performance -3-

Various conditions and I/O Performance