Recently I had to tackle a badly installed Solaris machine which hadn’t been configured with enough swap space. Luckily it had been built with a ZFS root filesystem, which made dealing with this a lot less painful.
First of all we need to get the details of our current swap setup:
bash-3.00# swap -l
swapfile dev swaplo blocks free
/dev/zvol/dsk/rpool/swap 256,2 16 4194288 4194288
New step is to increase the size of the ZFS ‘filesystem’ under the root pool (here called the default, rpool).
bash-3.00# zfs set volsize=4G rpool/swap
Once the filesystem size has been increased, we need to actually add it as swap. The normal swap command will do this – we just need to make sure we’re pointing it at the correct ZFS device:
bash-3.00# env NOINUSE_CHECK=1 swap -a /dev/zvol/dsk/rpool/swap $((8+4194288))
Let’s just check the status via ZFS:
bash-3.00# zfs list rpool/swap
NAME USED AVAIL REFER MOUNTPOINT
rpool/swap 4G 3.16G 2.03G -
And finally we can see the new swap space we’ve just added:
bash-3.00# swap -l
swapfile dev swaplo blocks free
/dev/zvol/dsk/rpool/swap 256,2 16 4194288 4194288
/dev/zvol/dsk/rpool/swap 256,2 4194304 4194304 4194304
A simple handful of commands, and no downtime – adding extra swap space using ZFS on Solaris is pretty painless. In another post I’ll explore how to grow ZFS filesystems like /var.
Last week Oracle held a marathon 5 hour webcast session, where they laid out their plans for Sun and their technologies. Sun’s website now redirects to Oracle, and although all the old Sun website links are still live, it’s now Oracle through and through.
The webcast held no surprises, really. As I mentioned previously about the Sun/Oracle merger and Larry’s talk on Oracle’s use of Sun technology, Oracle weren’t going to ditch Sun’s hardware line. The analysts were full of hysterics and gloom, but I’ve yet to meet an analyst who has the slightest clue of what’s going on. They’re paid to make noise and sell ‘research’, not to know what they’re talking about.
As predicted, there’s more investment in Sun’s hardware line, including lots more tasty new CMT processors, and a scaling up of the line to larger multi-socket machines. The high end gear will continue, as will the partnership with Fujitsu. SPARC continues to get a lot of investment and love, and will be a big focus going forwards. Amen to that.
Pretty much all of the software stack will stay and get integrated with Oracle’s offerings. I note with distaste that Oracle’s crappy Internet Directory remains the ‘enterprise’ offering for LDAP and identity management, with Sun’s LDAP products being pushed at smaller deployments. On the OS side, it’s a bit of Linux, and Solaris, Solaris, Solaris – Oracle recognise it’s the best commercial UNIX currently on the market, and that the feature set is unmatched.
Storage lives on, with Sun’s excellent Amber Road Storge 7000 Unified Storage boxes becoming ZFS appliances. Particularly exciting is their integration into OEM – imagine simple management of RMAN backups to ZFS appliances, giving low level snapshots and all sorts of goodness. I can see a lot of places going for that in a big way.
The big question for me was around OpenSolaris. No mention of it at all. It’s Open Source – that particular cat is out of the bag, and it’s not going away. So the question is what sort of effort will Oracle put behind it? Lots of new, OpenSolaris specific features – the new IPS packaging system and the Automated Installer – have potential, but aren’t up to scratch yet, and quite frankly don’t play well with existing Solaris infrastructure.
My bet is we’ll see less effort in re-inventing the wheel, and more focus on making OpenSolaris a more palatable Solaris 11. There’s a big Solaris installed base out there, and the focus on x86 and new features has so far meant that OpenSolaris isn’t really a credible upgrade path.
As I expected when I first heard the news, Oracle are going to be leveraging Sun’s technology and services and own and optimise the entire stack, from the silicon up to the application. This gives them a chance to really tune everything and to go head to head with IBM. May you live in interesting times, as they say.
Obviously there’s more, lots more. Oracle have handily posted up each section of the webcast so you can pick and choose which session you want to watch here. There are also a series of special short webcasts which focus on specific product areas – you can view them all here.
PS: as a side note, Thomas Kurian, who presented the Software Strategy webcast, managed to give one of the dullest presentations I’ve seen. Seriously, that was a really important session, but I almost nodded off a couple of times. Dire.
The idea behind processor sets has been around for a decade or so in the HPC arena. You’ve got certain jobs, that require a certain amount of CPU resources, or a certain IO profile, so you want to dedicate some CPUs just to them. Solaris has had processor controls in since the dark days of 2.6.
*Note:* I’m going to be freely talking about CPUs as the processing unit. This is all on T2ks and so I know that they’re not *real* CPUs – call them thread processing units or something, but for simplicity this document will just call them CPUs and be done with it.
The actual management of processor sets is very straightforward, and I’ll be playing about with them on one of my favourite bits of kit – the Sun T2000.
First of all we use the psrinfo command to view the status of our processors:
bash-3.00# psrinfo
0 on-line since 11/21/2006 20:24:57
1 on-line since 11/21/2006 20:24:58
2 on-line since 11/21/2006 20:24:58
3 on-line since 11/21/2006 20:24:58
4 on-line since 11/21/2006 20:24:58
5 on-line since 11/21/2006 20:24:58
6 on-line since 11/21/2006 20:24:58
7 on-line since 11/21/2006 20:24:58
8 on-line since 11/21/2006 20:24:58
9 on-line since 11/21/2006 20:24:58
10 on-line since 11/21/2006 20:24:58
11 on-line since 11/21/2006 20:24:58
12 on-line since 11/21/2006 20:24:58
13 on-line since 11/21/2006 20:24:58
14 on-line since 11/21/2006 20:24:58
15 on-line since 11/21/2006 20:24:58
16 on-line since 11/21/2006 20:24:58
17 on-line since 11/21/2006 20:24:58
18 on-line since 11/21/2006 20:24:58
19 on-line since 11/21/2006 20:24:58
20 on-line since 11/21/2006 20:24:58
21 on-line since 11/21/2006 20:24:58
22 on-line since 11/21/2006 20:24:58
23 on-line since 11/21/2006 20:24:58
24 on-line since 11/21/2006 20:24:58
25 on-line since 11/21/2006 20:24:58
26 on-line since 11/21/2006 20:24:58
27 on-line since 11/21/2006 20:24:58
28 on-line since 11/21/2006 20:24:58
29 on-line since 11/21/2006 20:24:58
30 on-line since 11/21/2006 20:24:58
31 on-line since 11/21/2006 20:24:58
Let’s do a quick network performance test with iperf to see what sort of throughput we can get when all processing units are able to process network IO:
bash-3.00# ./iperf --client np1unx0006 --time 60 --dualtest
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 48.0 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to np1unx0006, TCP port 5001
TCP window size: 48.0 KByte (default)
------------------------------------------------------------
[ 5] local 192.168.105.62 port 37438 connected with 192.168.105.59 port 5001
[ 4] local 192.168.105.62 port 5001 connected with 192.168.105.59 port 63459
[ 5] 0.0-60.0 sec 3.77 GBytes 540 Mbits/sec
[ 4] 0.0-60.0 sec 3.62 GBytes 518 Mbits/sec
At the same time, let’s have a look with mpstat to get an idea of what the processors are dealing with while this is going on.
The important colums here are intr, showing the amount of interrupts each CPU is handling. We also need to keep an eye on the number of system calls each CPU is fielding (syscl) and also the context switches and involuntary context switches (csw and icsw respectively) to make sure jobs are completely before the scheduler kicks them off the CPU.
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 96 0 248 3481 0 7126 4 34 439 0 6041 2 25 0 74
1 122 0 171 1332 0 2796 2 24 340 0 2369 2 14 0 85
2 79 0 216 646 0 1472 0 18 226 0 202 0 5 0 95
3 30 0 143 356 0 829 0 16 137 0 23 0 2 0 98
4 47 0 260 618 0 1514 0 18 163 0 74 0 3 0 97
5 56 0 257 714 0 1662 1 19 234 0 311 1 6 0 94
6 67 0 466 1085 0 2593 1 19 588 0 1234 0 17 0 82
7 26 0 268 894 0 2031 0 18 202 0 136 0 4 0 96
8 241 0 341 993 0 2286 0 22 258 0 358 1 7 0 91
9 190 0 292 1431 0 3102 1 21 257 0 1551 1 9 0 90
10 114 0 336 1155 0 2580 0 18 286 0 429 0 6 0 94
11 28 0 283 837 0 1883 1 18 551 0 1283 1 15 0 84
12 0 0 1 2 0 3 0 1 0 0 0 0 0 0 100
13 0 0 1 3 0 4 0 1 1 0 1 0 0 0 100
14 3 0 2 5 0 9 0 1 4 0 3 0 0 0 100
15 0 0 0 9 0 0 8 0 4 0 534955 75 25 0 0
16 64 0 423 1299 110 2418 0 18 286 0 59 0 5 0 95
17 89 0 454 1473 0 3233 0 19 319 0 793 1 7 0 92
18 46 0 397 960 1 2217 0 18 290 0 39 0 4 0 96
19 79 0 321 1048 2 2340 2 19 494 0 2073 2 15 0 83
20 79 0 205 852 1 1773 1 21 313 0 1493 1 14 0 85
21 27 0 19965 41259 41036 635 15 28 2862 0 415 0 47 0 53
22 65 0 129 1069 0 2274 1 21 139 0 1053 1 7 0 92
23 62 0 134 681 0 1446 1 20 370 0 931 1 14 0 85
24 115 0 260 799 0 1986 0 22 212 0 313 0 4 0 95
25 113 0 273 962 1 2225 1 22 266 0 684 1 7 0 93
26 73 0 312 1241 0 2862 0 23 271 0 663 0 6 0 94
27 115 0 270 862 0 2017 0 22 201 0 209 1 5 0 95
28 179 0 225 689 0 1548 0 17 213 0 302 1 5 0 94
29 42 0 224 656 0 1507 0 15 163 0 134 0 3 0 97
30 40 0 298 774 0 1821 1 14 459 0 1316 1 17 0 83
31 27 0 227 649 0 1544 1 15 644 0 1418 1 18 0 82
From this we can see we’re getting fairly decent throughput over GigE, and that the interrupts are spread across all the CPUs.
Now let’s create a processor set, and stick half our CPUs in it.
The command is psrset with the -c option to create a set. As this is the first processor set it will be processor set 1 – the next would be 2, etc. etc.
Remember we can get the number of our CPUs from the psrinfo command.
bash-3.00# psrset -c 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
created processor set 1
processor 0: was not assigned, now 1
processor 1: was not assigned, now 1
processor 2: was not assigned, now 1
processor 3: was not assigned, now 1
processor 4: was not assigned, now 1
processor 5: was not assigned, now 1
processor 6: was not assigned, now 1
processor 7: was not assigned, now 1
processor 8: was not assigned, now 1
processor 9: was not assigned, now 1
processor 10: was not assigned, now 1
processor 11: was not assigned, now 1
processor 12: was not assigned, now 1
processor 13: was not assigned, now 1
processor 14: was not assigned, now 1
processor 15: was not assigned, now 1
Now that we’ve assigned half our CPUs to processor set 1, we want to disable interrupt handling for them. We could use the psradm command to do it on a per CPU basis, but it’s much easier to just apply the setting to the entire processor set.
bash-3.00# psrset -f 1
The -f option disables interrupt handling, and the 1 is the processor set we want to apply this to.
We can check the effect by calling psrinfo again:
bash-3.00# psrinfo
0 no-intr since 12/19/2006 18:15:15
1 no-intr since 12/19/2006 18:15:15
2 no-intr since 12/19/2006 18:15:15
3 no-intr since 12/19/2006 18:15:15
4 no-intr since 12/19/2006 18:15:15
5 no-intr since 12/19/2006 18:15:15
6 no-intr since 12/19/2006 18:15:15
7 no-intr since 12/19/2006 18:15:15
8 no-intr since 12/19/2006 18:15:15
9 no-intr since 12/19/2006 18:15:15
10 no-intr since 12/19/2006 18:15:15
11 no-intr since 12/19/2006 18:15:15
12 no-intr since 12/19/2006 18:15:15
13 no-intr since 12/19/2006 18:15:15
14 no-intr since 12/19/2006 18:15:15
15 no-intr since 12/19/2006 18:15:15
16 on-line since 11/21/2006 20:24:58
17 on-line since 11/21/2006 20:24:58
18 on-line since 11/21/2006 20:24:58
19 on-line since 11/21/2006 20:24:58
20 on-line since 11/21/2006 20:24:58
21 on-line since 11/21/2006 20:24:58
22 on-line since 11/21/2006 20:24:58
23 on-line since 11/21/2006 20:24:58
24 on-line since 11/21/2006 20:24:58
25 on-line since 11/21/2006 20:24:58
26 on-line since 11/21/2006 20:24:58
27 on-line since 11/21/2006 20:24:58
28 on-line since 11/21/2006 20:24:58
29 on-line since 11/21/2006 20:24:58
30 on-line since 11/21/2006 20:24:58
31 on-line since 11/21/2006 20:24:58
Rock on! psrinfo clearly shows that half our CPUs will no longer handle interrupts. Let’s kick off another iperf throughput test and see what happens:
bash-3.00# ./iperf --client np1unx0006 --time 60 --dualtest
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 48.0 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to np1unx0006, TCP port 5001
TCP window size: 48.0 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.105.62 port 37419 connected with 192.168.105.59 port 5001
[ 5] local 192.168.105.62 port 5001 connected with 192.168.105.59 port 63457
[ 4] 0.0-60.0 sec 3.36 GBytes 481 Mbits/sec
[ 5] 0.0-60.0 sec 3.05 GBytes 436 Mbits/sec
Looking at mpstat we can clearly see the effects:
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
2 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
3 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
4 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
5 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
6 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
7 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
8 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
9 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
10 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
11 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
12 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
13 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
14 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
15 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
16 336 0 276 1854 112 3372 12 49 844 0 135008 24 37 0 39
17 261 0 115 2236 1 4594 6 51 1050 0 27470 7 37 0 56
18 104 0 105 1699 3 3506 9 36 1071 0 103500 17 38 0 44
19 106 0 148 855 1 1800 2 28 286 0 10881 2 8 0 89
20 290 0 19102 42492 42200 838 28 42 2676 0 74629 17 60 0 23
21 256 0 801 1952 0 4397 5 39 1272 0 2196 2 29 0 68
22 209 0 475 1191 0 2663 2 38 552 0 776 1 12 0 87
23 260 0 500 1134 4 2540 2 38 597 0 13071 4 13 0 84
24 455 0 752 2213 1 5038 4 41 916 0 10316 4 20 0 77
25 500 0 803 2485 0 5499 4 45 1352 0 17171 5 31 0 64
26 654 0 683 1773 0 4009 5 45 933 0 2119 8 19 0 73
27 503 0 516 1812 0 3952 5 45 748 0 21682 6 16 0 79
28 552 0 860 2332 0 5093 7 40 1065 0 12217 16 21 0 63
29 480 0 688 2292 0 4996 4 47 924 0 1395 3 17 0 80
30 663 0 476 1553 0 3357 5 45 658 0 2680 9 16 0 75
31 485 0 445 1520 0 3297 4 47 716 0 1167 2 16 0 82
We can see the non-interrupt handling CPUs in processor set 1 are totally idle – they’re just sitting there, twiddling their thumbs, and laughing at the other 16 CPUs working their socks off.
Involuntary context switches aren’t causing us an issue, so we can see that even with the reduced number of CPUs handling the interrupts, they’re still managed to deal with the load.
Now let’s see what happens when we execute the single-thread iperf process inside processor set 1. We can control this by using the psrset command to launch our app.
bash-3.00# psrset -e 1 ./iperf --client np1unx0006 --time 60 --dualtest
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 48.0 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to np1unx0006, TCP port 5001
TCP window size: 48.0 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.105.62 port 37419 connected with 192.168.105.59 port 5001
[ 5] local 192.168.105.62 port 5001 connected with 192.168.105.59 port 63457
[ 4] 0.0-60.0 sec 3.36 GBytes 481 Mbits/sec
[ 5] 0.0-60.0 sec 3.05 GBytes 436 Mbits/sec
And mpstat should give us an idea of what’s happening:
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 0 0 0 7751 0 16048 3 0 179 0 15235 3 42 0 55
1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
2 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
3 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
4 0 0 0 201 0 403 6 0 2478 0 8254 3 81 0 16
5 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
6 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
7 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
8 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
9 0 0 1 3 0 4 0 0 2 0 0 0 0 0 100
10 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
11 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
12 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
13 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
14 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
15 0 0 0 8 0 0 7 0 4 0 532794 75 25 0 0
16 313 0 942 1346 114 2533 1 20 459 0 419 1 13 0 86
17 306 0 450 687 1 1529 0 13 297 0 558 1 13 0 86
18 399 0 467 653 3 1442 1 14 255 0 3559 13 13 0 74
19 221 0 326 509 1 1164 0 11 196 0 401 1 7 0 92
20 646 0 10825 47201 47171 90 7 18 2405 0 1210 4 55 0 40
21 156 0 673 757 0 1769 0 15 346 0 201 1 9 0 91
22 220 0 959 1055 0 2467 0 15 488 0 397 2 12 0 86
23 204 0 844 791 2 1844 0 14 443 0 223 1 12 0 88
24 341 0 718 1033 1 2439 1 13 401 0 2571 10 14 0 76
25 205 0 570 804 0 1945 0 13 314 0 376 1 9 0 90
26 262 0 369 584 0 1379 0 12 204 0 422 1 7 0 92
27 199 0 348 519 0 1226 0 13 180 0 533 2 6 0 92
28 356 0 434 726 0 1730 0 17 247 0 515 2 9 0 89
29 393 0 267 428 0 1043 1 18 197 0 999 3 11 0 86
30 367 0 491 812 0 1829 0 17 298 0 449 1 10 0 89
31 302 0 424 675 0 1538 0 15 245 0 339 1 8 0 91
Well, that’s broken things. How come the processors in the set are now handling interrupts?
It looks like executing the binary inside the processor set still generates interrupts – but these are unlikely to be network I/O. Check out the number of syscalls being generated! It’s likely an artefact of my poor choice of application – iperf generates a huge amount of interrupts and can really cane your ethernet interfaces.
We could use dtrace to have a real poke around, but I think that should be the topic for another day.
Now we’ve finished playing around, we need to re-enable interrupt handling on those CPUs. As the -f flag to psrset disabled interrupt handling, -n is the option we need to re-enabled interrupt handling on a processor set.
bash-3.00# psrset -n 1
Now the CPUs are handling interrupts again, we need to delete the processor set. We do this by passing the psrset command the -d option, and giving it the processor set number:
bash-3.00# psrset -d 1
removed processor set 1
Finally let’s run psrinfo and double check the state of our CPUs:
bash-3.00# psrinfo
0 on-line since 12/19/2006 18:23:42
1 on-line since 12/19/2006 18:23:42
2 on-line since 12/19/2006 18:23:42
3 on-line since 12/19/2006 18:23:42
4 on-line since 12/19/2006 18:23:42
5 on-line since 12/19/2006 18:23:42
6 on-line since 12/19/2006 18:23:42
7 on-line since 12/19/2006 18:23:42
8 on-line since 12/19/2006 18:23:42
9 on-line since 12/19/2006 18:23:42
10 on-line since 12/19/2006 18:23:42
11 on-line since 12/19/2006 18:23:42
12 on-line since 12/19/2006 18:23:42
13 on-line since 12/19/2006 18:23:42
14 on-line since 12/19/2006 18:23:42
15 on-line since 12/19/2006 18:23:42
16 on-line since 11/21/2006 20:24:58
17 on-line since 11/21/2006 20:24:58
18 on-line since 11/21/2006 20:24:58
19 on-line since 11/21/2006 20:24:58
20 on-line since 11/21/2006 20:24:58
21 on-line since 11/21/2006 20:24:58
22 on-line since 11/21/2006 20:24:58
23 on-line since 11/21/2006 20:24:58
24 on-line since 11/21/2006 20:24:58
25 on-line since 11/21/2006 20:24:58
26 on-line since 11/21/2006 20:24:58
27 on-line since 11/21/2006 20:24:58
28 on-line since 11/21/2006 20:24:58
29 on-line since 11/21/2006 20:24:58
30 on-line since 11/21/2006 20:24:58
31 on-line since 11/21/2006 20:24:58
Solaris processor sets are the easiest to use of all the resource controls built into the OS. We can peg things like zones, individual applications, or even specific processes, to their own processor sets to control and manage resource usage. This gives us some really fine grained control over how the system is used, and with a machine like the T2000 it allows us to really scale performance.
Logfile management has long been a bane for sysadmins everywhere. Applications seem to scatter logfiles all over the place, and they grow at an alarming rate. We want the information in them, so we need to cycle and compress them. Previously this involved writing custom scripts that can handle the logfile management and restarting the application – and to add to the pain, these scripts had to be tested, deployed, and monitored.
Luckily Solaris comes with a handy utility called logadm, which is used by the OE to manage some of the core system log files. logadm can quickly and easily be used to handle all of our log file managment needs.
Let’s look at two log files which aren’t handled by Solaris out of the box – sulog and wtmpx. Both are important, as they help us with our user access audit trail. For starters, we want to keep two old copies of each, and we want to cycle them every two weeks.
Our logadm syntax looks like this:
/usr/sbin/logadm -C 2 -p 2w -c -w <full_path_to_logfile>
- -C number of copies to keep
- -p time between each log cycle (2 weeks)
- -c copy and them truncate (doesn’t need to restart a service then)
- -w writes an entry into /etc/logadm.conf for this log file
So executing the following:
bash-3.00# logadm -C 2 -p 2w -c -w /var/adm/sulog
bash-3.00# logadm -C 2 -p 2w -c -w /var/adm/wtmpx
will result in the following two lines being written to the end of /etc/logadm.conf
/var/adm/wtmpx -C 2 -c -p 2w
/var/adm/sulog -C 2 -c -p 2w
Getting logadm to add an entry to /etc/logadm.conf means that this won’t be a one-off thing – each time logadm executes from cron, it will read the entries from this file. Each entry is checked to see if the log file’s size or age means it’s due for rotation.
wtmpx is obviously a binary file that’s used by last – rather than having to restart the utmpd daemon, it’s easier to just truncate the file. last can still read the older copies – just use the syntax
last -f <wtmpx_file>
It’s important to properly cycle wtmpx, rather than just deleting or truncating it, because it provides a helpful audit trail of users who accessed the system – showing when they logged in, and from where.
This is great if we just want to cycle logs around – but what if we want to compress them as well? Apache is the poster child for log generation – it spits out copious amounts of data, and you want to keep it all for analysis, but it’s a pain to manage.
On my test machine I’ve deployed Apache via Blastwave, so it’s logging to /opt/csw/apache2/var
With SSL enabled there are five log files that I’m interested in:
bash-3.00# ls -l
total 31536
-rw-r--r-- 1 root other 5400214 Oct 30 16:26 access_log
-rw-r--r-- 1 root other 8716843 Oct 29 23:18 error_log
-rw-r--r-- 1 root root 760298 Oct 30 16:26 ssl_access_log
-rw-r--r-- 1 root root 268873 Oct 30 16:26 ssl_error_log
-rw-r--r-- 1 root root 934541 Oct 30 16:26 ssl_request_log
We can just modify the previous logadm command to handle wtmpx – but what about the compression? Helpfully logadm will automatically compress cycled log files using gzip, if we pass it the -z flag. -z will also take a count option, which tells logadm to leave the most recent logfiles uncompressed.
In this case, however, we want everything except the current in-flight log file compressed, and we want to cycle when the logfile reaches 10mb is size:
bash-3.00# logadm -C 10 -s 10m -c -z 0 -w /opt/csw/apache2/var/log/access_log
logadm drops an entry into /etc/logadm.conf for us:
/opt/csw/apache2/var/log/access_log -C 10 -c -s 10m -z 0
Add an entry for each of the five log files, and we end up with this in /etc/logadm.conf:
/opt/csw/apache2/var/log/access_log -C 10 -c -s 10m -z 0
/opt/csw/apache2/var/log/error_log -C 10 -c -s 10m -z 0
/opt/csw/apache2/var/log/ssl_access_log -C 10 -c -s 10m -z 0
/opt/csw/apache2/var/log/ssl_error_log -C 10 -c -s 10m -z 0
/opt/csw/apache2/var/log/ssl_request_log -C 10 -c -s 10m -z 0
Using a combination of log cycling and compression, logadm can handle pretty much any application’s log files for us. By using copy and truncate as well, we aren’t forced to restart each application when we cycle the logs, which ends up giving us a huge amount of control over our log files, without having to write and maintain shell scripts.
We’ve all been there – there’s a missing binary or library on a Solaris host, someone’s accidentally deleted it, and we need to re-install the package. Or the other common scenario – two boxes that have been built by hand, one has binary A and the other doesn’t – which package adds it?
Along with the usual Solaris package management commands of pkgadd and pkgrm, there’s a lesser known utility called pkgchk. pkgchk allows us to check which package a file belongs to.
pkgchk will work with binaries:
-bash-3.00$ /usr/sbin/pkgchk -l -p /usr/bin/less
NOTE: Couldn't lock the package database.
Pathname: /usr/bin/less
Type: regular file
Expected mode: 0555
Expected owner: root
Expected group: bin
Expected file size (bytes): 117760
Expected sum(1) of contents: 20724
Expected last modification: Jan 23 01:48:30 2005
Referenced by the following packages:
SUNWless
Current status: installed
And we can also invoke pkgchk for libraries as well:
-bash-3.00$ /usr/sbin/pkgchk -l -p /lib/libresolv.so.1
NOTE: Couldn't lock the package database.
Pathname: /lib/libresolv.so.1
Type: regular file
Expected mode: 0755
Expected owner: root
Expected group: bin
Expected file size (bytes): 48368
Expected sum(1) of contents: 5063
Expected last modification: Jan 23 01:44:54 2005
Referenced by the following packages:
SUNWcslr
Current status: installed
For a quick overview of some of the other options, just invoke pkgchk with the -? command line:
-bash-3.00$ /usr/sbin/pkgchk -?
usage:
pkgchk [-l|vqacnxf] [-R rootdir] [-p path[, ...] | -P path[, ...]]
[-i file] [options]
pkgchk -d device [-f][-l|v] [-p path[, ...] | -P path[, ...]]
[-V ...] [-M] [-i file] [-Y category[, ...] | pkginst [...]]
where options may include ONE of the following:
-m pkgmap [-e envfile]
pkginst [...]
-Y category[, ...]
As you can see from the output, pkgchk is also very handy to see if the binaries or libraries that were part of a Solaris package file have been overwritten. As each file in a package has it’s checksum, file size, ownership, and permissions stored as part of the package manifest, this gets added to the Solaris package database when the package is installed – giving a quick and easy method to sanity check your installation.
