11-29-12 - Unnatural

I hate having neighbors so much. It's just not a natural way to live, this modern human way, where we're all crammed together with people who are not our tribe.

I believe that human beings are only comfortable living with people they are intimate with. In ancient days this was your whole tribe, now it's usually just family. You essentially have no privacy from these people, and not even separate property. Trying to keep your own stuff is an exercise in frustration. You must trust these people and work together and open up to them to be happy. Certainly there is always friction in this, but it's a natural human existence, and even though it may give you lots to complain about, there will also be joy. (foolishly moving way from this way of life is the root of much unhappiness)

Everyone else is an enemy. If you aren't in my intimate tribal group, WTF are you doing near my home? This is my land where I have my family, I will fucking jab you in the eye with a pointy stick.

I'm not really comfortable with "friends". So-called "friends" are not your friends; they will make fun of you behind your back, they will let you down when you need help. You can't ever really open up and admit things to them, you can't show your weaknesses, they will mock you for it or use your weaknesses against you. It's so awful to me the way normal people talk to each other; everyone is pretending to be strong and happy all the time, nobody ever talks about anything serious, some people put on a big show to be entertaining, it's all just so tense and shallow and unpleasant. The reason is that these people are not in my tribe, hence they are my enemies, and this whole "friends" thing is a very modern societal invention that doesn't really work.

I realized a while ago that this is one of the things I hate about going into the office. The best office experiences I've had have been the ones where it was a small team, we were all young and spent lots of time at work, and we actually bonded and had fun together and were almost like a family after several years of crunching (going through tough shit together is a classic way to brainwash people into acting like a tribe); at that point, it feels comfortable to go in to work, you can rip a fart and people laugh instead of tensely pretending that nothing happened. But most of the time an office never reaches that level of intimacy, the people in the office are just acquaintances, so you're in this space where you're supposed to be relaxed, and there are people walking around all the time looking over your shoulder, but they are enemies! Of course I can't relax, they're not my tribe, why are they in my space? It's terrible.

Going away from home to work is really unnatural.

At first when people start working from home it feels weird because they're so used to leaving, but really this whole going to a factory/office thing is a super-modern invention (last few hundred years).

Of course you should stay home and work your farm. You should build your house, till your field, and be there to protect your women and children (eg. in the modern world : open jars for them). Of course you should have your children beside you so that you can talk to them and teach them your trade as you work.

Of course when you're hungry you should go in to your own kitchen and eat some braised pork shoulder that's real simple hearty food cooked by your own hands, not the poisonous filth that restaurants purvey.

You shouldn't leave your family for 8 hours every day, that's bizarre and horrible. You should see your livestock running around, be there to shoo away the neighbors cats, see the trees changing color, and put your time and your love into what is really yours.


11-28-12 - SSD Buying Guide

Have done a bunch of reading over the past 24 hours and updating my old posts on SSD's :

cbloom's definitive SSD buying guide :

recommended :

Intel's whatever (currently the 520, but actually the old X25-M is still just fine; the S3700 stuff looks promising for the future)

not recommended :

Everything else.

The whole issue of flash degradation and moving blocks and such is a total red herring. SSD's are not failing because of the theoretical lifetime of flash memory, they are failing because the non-Intel drives are just broken. It's pretty simple, don't buy them.

The other issue I really don't care about is speed. They're all super fast. If they all actually worked then maybe I would care which was fastest, but since the non-Intel ones are just broken, the question of speed is irrelevant. The hardware review sites are all pretty awful with their endless benchmarking and complete missing of the actual issues. And even my ancient X25-M is plenty fast enough.

I think it's tempting to just go for the enterprise-grade stuff (Intel 710 at the moment). Saving money on storage doesn't make any sense to me, and all the speed measurement stuff just makes me yawn. (Intel 720 looks promising for the future). It's not quite as clear cut as ECC RAM (which is obviously worth it), but I suspect that spending another few $hundred to not worry about drive failure is worth it.

Oh, also brief googling indicates various versions of Mac OS don't support various SSD's correctly. I would just avoid SSD's on Mac unless you are very confident about getting this right. (best practice is probably just avoiding Mac altogether, but YMMV and various other acronyms)


11-26-12 - VAG sucks

(VAG = Volkswagen Auto Group)

Going through old notes I found this (originally from Road and Track) :

"For instance, just about every Audi, Porsche and Volkswagen model that I've driven in the U.S. doesn't allow throttle/brake overlap. Our long-term Nissan 370Z doesn't, either, which is a big reason why I'm not particularly keen on taking it out for a good flog; overlap its throttle and brake just a little bit and the Z cuts power for what seems an eternity (probably about two seconds)."

VAG makes fucked up cars. I certainly won't ever buy a modern one again. They have extremely intrusive computers that take the power for LOLs out of the driver's hands. (apparently the 370Z also has some stupidity as well; this shit does not belong in cars that are sold as "driver's cars").

(in case it wasn't clear from the above : you cannot left-foot-brake a modern Porsche with throttle overlap. Furthermore, you also can't trail-brake oversteer a modern Porsche because ESC is always on under braking. You have to be careful going fully off throttle and then back on due to the off-throttle-timing-advance. etc. etc. probably more stupid shit I'm not aware of. This stuff may be okay for most drivers in their comfort saloons, but is inexcusable in a sports car)

Anyway, I'm posting cuz this reminded me that I found another good little mod for the 997 :

Stupid VAG computer has clutch-release-assist. What this does is change the engine map in the first few seconds after you let the clutch out. The reason they do this is so that incompetent old fart owners don't stall the car when pulling away from a light, and also to help you not burn the clutch so much. (the change to the engine map increases the minimum throttle and also reduces the max).

If you actually want to drive your car and do hard launches and clutch-kicks and generally have fun, it sucks. (the worst part is when you do a hard launch and turn, like when you're trying to join fast traffic, and you get into a slight slide, which is fine and fun, but then in the middle of your maneuver the throttle map suddenly changes back as the clutch-assist phase ends, and the car sort of lurches and surges weirdly, it's horrible). Fortunately disabling it is very easy :

There's a sensor that detects clutch depression. It's directly above the clutch in the underside of the dash. You should be able to see the plastic piston for the sensor near the hinge of the clutch pedal. All you have to do is unplug the sensor (it's a plastic clip fitting)

With the sensor unplugged you get no more clutch-release-assist and the car feels much better. You will probably stall it a few times as you get used to the different throttle map, but once you're used to it smooth fast starts are actually easier. (oh, and pressing the clutch will no longer disable cruise control, so be aware of that). I like it.

(aside : it's a shame that all the car magazines are such total garbage. If they weren't, I would be able to find out if any modern cars are not so fucked. And you also want to know if they're easy to fix; problems that are easy to fix are not problems)

(other aside : the new 991-gen Cayman looks really sweet to me, but there are some problems. I was hoping they would use the longer wheelbase to make the cabin a bit bigger, which apparently they didn't really do. They also lowered the seat and raised the door sills which ruin one of the great advantages of the 997-gen Porsches (that they had not adopted that horrible trend of excessively high doors and poor visibility). But the really big drawback is that I'm sure it's all VAG-ed up in stupid ways that make it annoying for a driver. And of course all the standard Cayman problems remain, like the fact that they down-grade all the parts from the 911 in shitty ways (put the damn multi-link rear suspension on the Cayman you assholes, put an adjustable sway on it and adjustable front control arms))

(final aside : car user interface design is generally getting worse in the last 10-20 years. Something that user interface designers used to understand but seem to have forgotten is that the most potent man-machine bond develops when you can build muscle memory for the device, so that you can use it effectively with your twitch reflexes that don't involve rational thought. In order for that to work, the device must behave exactly the same way at all times. You can't have context-sensitive knobs. You can't have the map of the throttle or brake pedal changing based on something the car computer detected. You must have the same outcome from the same body motion every time. This must be an involiable principle of good user interfaces.)

11-26-12 - Chickens and Hawks

Hawk with kill in our yard :

And in context :

The chickens were out free ranging at the time; they all ran inside the coop and climbed back into the farthest corner nesting box and sat on top each other in a writhing pile of terrified chickens.

Watching animals is pretty entertaining. I remember when I was younger, I used to think it was a pathetic waste of time. Old people would sit around and watch the cats play, or get all excited about going on safari or whatever, and I would think "ppfft, boring, whatever, I've seen it on TV, what a sad vapid way to get entertainment, you oldsters are all so brain-dead, doing nothing with your time, you could be learning quantum field theory, but you've all just given up on life and want to sit around smiling at animals". Well, that's me now.


11-24-12 - The Adapted Eye

Buying a house for a view is a big mistake. (*)

Seattle is a somewhat beautiful place (I'm not more enthusiastic because it is depressing to me how easily it could have been much better (and it continues to get worse as the modern development of Cap Hill and South Lake Union turn the city into a generic condo/mall dystopia)) but I just don't see it any more. When we got back from CA I realized that I just don't see the lake and the trees anymore, all I see is "home".

There are some aspects that still move me, like clear views of the Olympics, because they are a rare treat. But after 4 years, the beauty all around is just background.

We have pretty great views from our house, and I sort of notice them, but really the effect on happiness of the view is minimal.

(* = there are benefits to houses with a view other than the beauty of the view. Usually a good view is associated with being on a hill top, or above other people, or up high in a condo tower, and those have the advantages of being quieter, better air, more privacy, etc. Also having a view of nature is an advantage just in the fact that it is *not* a view of other people, which is generally stressful to look at because they are doing fucked up things that you can't control. I certainly appreciate the fact that our house is above everyone else; it's nice to look down on the world and be separate from it).

I was driving along Lake Wash with my brother this summer and he made some comment about how beautiful it was, and for a second there I just couldn't figure out what he was talking about. I was looking around to see if there was some new art installation, or if Mount Rainier was showing itself that day, and then I realized that he just meant the tree lined avenue on the lake and the islands and all that which I just didn't see at all any more.

Of course marrying for beauty is a similar mistake. Even ignoring the fact that beauty fades, if we imagine that it lasted forever it would still be a mistake because you would stop seeing it.

I've always thought that couples could keep the aesthetic interest in each other alive by completely changing their style every few years. Like, dress as a hipster for a while, dress as a punk rocker or a goth, dress as a preppy business person. Or get drastically different hair cuts, like for men grow out your hair like an 80's rocker, or get a big Morrisey pompadour, something different. Most people over 30 tend to settle into one boring low-maintenance style for the rest of their lives, and it becomes invisible to the adapted eyes in their lives.

I suppose there are various tricks you can use; like rather than have your favorite paintings on the wall all the time, rotate them like a museum, put some in storage for a while and hang up some others. It might even help to roll some dice to forcibly randomize your selection.

I guess the standard married custom of wearing sweats around the house and generally looking like hell is actually a smart way of providing intermittent reward. It's the standard sitcom-man refrain to complain that your wife doesn't fancy herself up any more, but that's dumb; if she did dress up every day, then that would just become the norm and you would stop seeing it. Better to set the baseline low so that you can occasionally have something exceptional.

(add : hmm the generalized point that you should save your best for just a few moments and be shitty other times is questionable. Think about behavior. Should you intentionally be kind of dicky most of the time and occasionally really nice? If you're just nice all the time, that becomes the baseline and people take it for granted. I'm not sure about that. But certainly morons do love the "dicky dad" character in TV's and movies; your typical fictional football coach is a great example; dicky dad is stern and tough, scowly and hard on you, but then takes you aside and is somewhat kind and generous, and all the morons in the audience melt and just eat that shit up.)

One of the traps of life is optimizing things. You paint your walls your favorite color for walls, you think you're making things better, but that gets you stuck in a local maximum, which you then stop seeing, and you don't feel motivated to change it because any change is "worse".

I realized the other day that quite a few ancient societies actually have pretty clever customs to provide randomized rewards. For example lots of societies have something like "numbers" , which ignoring the vig, is just a way of taking a steady small income and turning it into randomized big rewards.

Say you got a raise and make $1 more a day. At first you're happy because your life got better, but soon that happiness is gone because you just get used to the new slightly better life and don't perceive it any more. If instead of getting that $1 a day, you instead get $365 randomly on average once a year, your happiness baseline is the same, but once in a while you get a really happy day. This is probably actually better for happiness.

I think the big expensive parties that lots of ancient societies throw for special events might be a similar thing. Growing up in LA we would see our poor latino neighbors spend ridiculous amounts on a quincenera or a wedding and think how foolish it was, surely it's more rational to save that money and use it for health care or education or a nicer house. But maybe they had it right? Human happiness is highly resistant to rational optimization.


11-23-12 - Global State Considered Harmful

In code design, a frequent pattern is that of singleton state machines. eg. a module like "the log" or "memory allocation" which has various attributes you set up that affect its operation, and then subsequent calls are affected by those attributes. eg. things like :

Log_SetOutputFile( FILE * f );


Log_Printf( const char * fmt .... );

or :

malloc_setminimumalignment( 16 );


malloc( size_t size );

The goal of this kind of design is to make the common use API minimal, and have a place to store the settings (in the singleton) so they don't have to be passed in all the time. So, eg. Log_Printf() doesn't have to pass in all the options associated with logging, they are stored in global state.

I propose that global state like this is the classic mistake of improving the easy case. For small code bases with only one programmer, they are mostly okay. But in large code bases, with multi-threading, with chunks of code written independently and then combined, they are a disaster.

Let's look at the problems :

1. Multi-threading.

This is an obvious disaster and pretty much a nail in the coffin for global state. Say you have some code like :

pcb * previous_callback = malloc_setfailcallback( my_malloc_fail_callback );

void * ptr = malloc( big_size ); 

malloc_setfailcallback( previous_callback );

this is okay single threaded, but if other threads are using malloc, you just set the "failcallback" for them as well during that span. You've created a nasty race. And of course you have no idea whether the failcallback that you wanted is actually set when you call malloc because someone else might change it on another thread.

Now, an obvious solution is to make the state thread-local. That fixed the above snippet, but some times you want to change the state so that other threads are affected. So now you have to have thread-local versions and global versions of everything. This is a viable, but messy, solution. The full solution is :

There's a global version of all state variables. There are also thread-local copies of all the global state. The thread-local copies have a special value that means "inherit from global state". The initial value of all the thread-local state should be "inherit". All state-setting APIs must have a flag for whether they should set the global state or the thread-local state. Scoped thread-local state changes (such as the above example) need to restore the thread-local state to "inherit".

This can be made to work (I'm using for the Log system in Oodle at the moment) but it really is a very large conceptual burden on the client code and I don't recommend it.

There's another way that these global-state singletons are horrible for multi-threading, and that's that they create dependencies between threads that are not obvious or intentional. A little utility function that just calls some simple functions picks up these ties to shared variables and needs synchronization protection with the global state. This is related to :

2. Non-local effects.

The global state makes the functions that use it non-"pure" in a very hidden way. It means that innocuous functions can break code that's very far away from it in hidden ways.

One of the classic disasters of global state is the x87 (FPU) control word. Say you have a function like :

void func1()

    set x87 CW

    do a bunch of math that relies on that CW


    do more math that relies on CW

    restore CW

Even without threading problems (the x87 CW is thread-local under any normal OS), this code has nasty non-local effects.

Some branch of code way out in func2() might rely on the CW being in a certain state, or it might change the CW and that breaks func1().

You don't want to be able to break code very far away from you in a hidden way, which is what all global state does. Particularly in the multi-threaded world, you want to be able to detect pure functions at a glance, or if a function is not pure, you need to be able to see what it depends on.

3. Undocumented and un-asserted requirements.

Any code base with global state is just full of bugs waiting to happen.

Any 3d graphics programmer knows about the nightmare of the GPU state machine. To actually write robust GPU code, you have to check every single render state at the start of the function to ensure that it is set up the way you expect. Good code always expresses (and checks) its requirements, and global state makes that very hard.

This is a big problem even in a single-source code base, but even worse with multiple programmers, and a total disaster when trying to copy-paste code between different products.

Even something like taking a function that's called in one spot in the code and calling it in another spot can be a hidden bug if it relied on some global state that was set up in just the right way in that original spot. That's terrible, as much as possible functions should be self-contained and work the same no matter where they are called. It's sort of like "movement of call site invariance symmetry" ; the action of a function should be determined only by its arguments (as much as possible) and any memory locations that it reads should be as clearly documented as possible.

4. Code sharing.

I believe that global state is part of what makes C code so hard to share.

If you take a code snippet that relies on some specific global state out of its content and paste it somewhere else, it no longer works. Part of the problem is that nobody documents or checks that the global state they need is set. But a bigger issue is :

If you take two chunks of code that work independently and just link them together, they might no longer work. If they share some global state, either intentionally or accidentally, and set it up differently, suddenly they are stomping on each other and breaking each other.

Obviously this occurs with anything in stdlib, or on the processor, or in the OS (for example there are lots of per-Process settings in Windows; eg. if you take some libraries that want a different time period, or process priority class, or priviledge level, etc. etc. you can break them just by putting them together).

Ideally this really should not be so. You should be able to link together separate libs and they should not break each other. Global state is very bad.

Okay, so we hate global state and want to avoid it. What can we do? I don't really have the answer to this because I've only recently come to this conclusion and don't have years of experience, which is what it takes to really make a good decision.

One option is the thread-local global state with inheritance and overrides as sketched above. There are some nice things about the thread-local-inherits-global method. One is that you do still have global state, so you can change the options somewhere and it affects all users. (eg. if you hit 'L' to toggle logging that can change the global state, and any thread or scope that hasn't explicitly sets it picks up the global option immediately).

Other solutions :

1. Pass in everything :

When it's reasonable to do so, try to pass in the options rather than setting them on a singleton. This may make the client code uglier and longer to type at first, but is better down the road.

eg. rather than

malloc_set_alignment( 16 );

malloc( size );

you would do :

malloc_aligned( size , 16 );

One change I've made to Oodle is taking state out of the async systems and putting in the args for each launch. It used to be like :

OodleWork_SetKickImmediate( OodleKickImmediate_No );
OodleWork_SetPriority( OodlePriority_High );
OodleWork_Run( job );

and now it's :

OodleWork_Run( job , OodleKickImmediate_No, OodlePriority_High );

2. An options struct rather than lots of args.

I distinguish this from #3 because it's sort of a bridge between the two. In particular I think of an "options struct" as just plain values - it doesn't have to be cleaned up, it could be const or made with an initializer list. You just use this when the number of options is too large and if you frequently set up the options once and then use it many times.

So eg. the above would be :

OodleWorkOptions wopts = { OodleKickImmediate_No, OodlePriority_High  };
OodleWork_Run( job , &wopts );

Now I should emphasize that we already have given ourselves great power and clarity. The options struct could just be global, and then you have the standard mess with that. You could have it in the TLS so you have per-thread options. And then you could locally override even the thread-local options in some scope. Subroutines should take OodleWorkOptions as a parameter so the caller can control how things inside are run, otherwise you lose the ability to affect child code which a global state system has.

Note also that options structs are dangerous for maintenance because of the C default initializer value of 0 and the fact that there's no warning for partially assigned structs. You can fix this by either making 0 mean "default" for every value, or making 0 mean "invalid" (and assert) - do not have 0 be a valid value which is anything but default. Another option is to require a magic number in the last value of the struct; unfortunately this is only caught at runtime, not compile time, which makes it ugly for a library. Because of that it may be best to only expose Set() functions for the struct and make the initializer list inaccessible.

The options struct can inherit values when its created; eg. it might fill any non-explicitly given values (eg. the 0 default) by inheriting from global options. As long as you never store options (you just make them on the stack), and each frame tick you get back to a root for all threads that has no options on the stack, then global options percolate out at least once a frame. (so for example the 'L' key to toggle logging will affect all threads on the next frame).

3. An initialized state object that you pass around.

Rather than a global singleton for things like The Log or The Allocator, this idea is to completely remove the concept that there is only one of those.

Instead, Log or Allocator is a struct that is passed in, and must be used to do those options. eg. like :

void FunctionThatMightLogOrAllocate( Log * l, Allocator * a , int x , int y )
    if ( x )
        Log_Printf( l , "some stuff" );

    if ( y )
        void * p = malloc( a , 32 );

        free( a , p );

now you can set options on your object, which may be a per-thread object or it might be global, or it might even be unique to the scope.

This is very powerful, it lets you do things like make an "arena" allocator in a scope ; the arena is allocated from the parent allocator and passed to the child functions. eg :

void MakeSuffixTrie( Allocator * a , U8 * buf, int bufSize )

    Allocator_Arena arena( a , bufSize * 4 );

    MakeSuffixTrie_Sub( &arena, buf, bufSize );

The idea is there's no global state, everything is passed down.

At first the fact that you have to pass down a state pointer to use malloc seems like an excessive pain in the ass, but it has advantages. It makes it super clear in the signature of a function which subsystems it might use. You get no more surprises because you forgot that your Mat3::Invert function logs about degeneracy.

It's unclear to me whether this would be too much of a burden in real world large code bases like games.


11-13-12 - Manifesto on American Gardens

The landscaping in my neighborhood is so incredibly ugly, it occurred to me the other day that it would actually look better if everybody just stopped tending their yards for 5 years and let it go back to natural; some blackberry, some sweet peas, oregon grape, and conifers. Wild, bushy, green, that stuff is beautiful. The un-tended strips along the highways or in abandonded lots look better than the damn city, which is covered with all these Home Depot plants that don't belong in the area.

American vernacular garden style is disgusting. It consists of a patch of lawn that's generally very dense and tightly mowed, then a hard edge at the border of the lawn, often even with something utterly unforgivable like a black plastic strip, then planted beds. The planted bends typically are a big expanse of mulch with scattered little dots of isolated plants. It's totally unnatural looking; it doesn't look right in its place, like it belongs there.

Just like with food, you should ask yourself, is all this stuff I'm doing actually making it better? If you go out to the woods around here and then walk around a neighborhood, how could you possibly think that the concrete pavers and decks and inappropriate warm-weather plants are an improvement?

Anyhoo, here's my manifesto about natural garden design :

1. Plants should look like themselves. Ferns look good green, not yellow or variegated. Daffodils look good yellow, not red or white. These days with advanced hybridization techniques you can get all kinds of crazy stuff, but DON'T, they are tacky as hell. They're like heavy makeup or plastic surgery, they sort of optimize a beauty goal but wind up being worse.

2. Your garden should match your area. Again with careful tending you can grow things from all over the world, but you shouldn't. The plants will look the most natural if they suit your area. Here in the PNW that means evergreens, ferns, rhododendra, etc. You can add some Japanese plants and such that have similar native climates, but things like tropical plants or hot/dry mediterranean plants do not belong here.

3. Mulch is fucking disgusting looking. One of the worst possible things you can do to your garden is to spread a huge field of mulch and then dot it with a sparse scattering of shrubs. Mulch is a necessary evil (actually some modern thought believes that mulch is overrated, but that's a digression) and should be invisible as much as possible. Mulches should, like the plants, match the area, not be some weird imported thing. So the modern cocoa and coco (coconut) mulches are both inappropriate everywhere. Here in the Northwest, pine bark is a semi-natural forest floor mulch and so looks okay in moderation.

4. Fertilizers, weed killers, grass treatments, etc. are all massive poisons, they flow into our natural water ways and fuck up the environment. You are a huge fucking selfish asshole if you use them beyond the bare minimum that is absolutely necessary. If your yard or plants require large amounts of chemicals to be happy, then change your fucking yard you asshole, plant something that works better in your environment; poisoning the damn lake is not a good tradeoff for you having a nice lawn.

5. Moss is a very natural and beautiful part of a Northwest garden and should not be removed. (BTW on a semi-related topic, the idea that moss can damage your roof is basically a myth here in the northwest (we very rarely get a major freeze, which is what makes moss harmful (because freezing makes it expand which rips up the shingles)) - and certainly pressure washing is guaranteed to do more harm than the moss ever could). Removing moss in the northwest is like polishing the patina off antique metal ware, it shows a complete lack of taste.

6. Concrete, manufactured stone, plastic, cast pavers, etc. have no place in a garden. Landscape fabric, plastic path/yard edging, etc. can be used but only if they will stay invisible, which they won't (they always work themselves out into sight), so probably just shouldn't be used.

7. A sort of general issue I've been thinking about is that there is a conflict between what looks good in a photo, or in a first impression, vs. what looks good to live with. This is true of gardens, houses, lots of things. Basically for a photo or a first impression (like a realtor visit) what looks good is simple, clean, and above all coherent; for example flowers should be all of one color or two colors. Many people now design gardens with this in mind, optimizing for a single view. However, that is quite different from what is enjoyable to live with on a regular basis. The scene that looks great in a photo will get boring if it's all you have to look at every day. To live with it's nicer to have lots of variety, unusual specimens, lots of little bits of interest you can walk around and look at. It's sort of like the overall impression vs. the density of interest; it's the "macro" vs "micro" optimization if you like. (the same is true of house decor; magazines and realtors favor a very clean, unified, almost Japanese simple interior, but living in that is quite boring; it's more interesting to live in the very cluttered house full of curios and covered with paintings and photos that give you lots of little things to look at).

The ideal Seattle house should have big Doug Fir beams, a cedar shake roof, and a big fireplace made of natural boulders. There should be a stream on the property and french drains that route groundwater to the stream.

The ideal Seattle garden should be like a woodland meadow. Obviously you don't actually want a "house in the woods" actually because it's too dark, what you want is that feeling when you're walking through the dense woods and you get to a big meadow and suddenly the sun appears and there's this lovely clearing of grasses and flowers with trees all around.

An ideal Seattle garden should always include some big evergreen trees, since they are the true masters of this landscape. A forest garden around the evergreens could include rhodendra, ferns, blueberries, etc. A good Seattle garden should always include moss and boulders; a truly lucky site would have one of our magnificent ice-age glacially deposited boulders.

(Seattle used to have lots of amazing giant boulders in the city. They were deposited by the glaciers that cut the sound, and were usually granite, giant 40 foot diameter things that just plopped there randomly. The vast majority of them have been destroyed, clearing space for houses and roads and such, and also to create smaller rocks. If you drive around Seattle you may observe all the rockeries used as retaining walls, made of large boulders (2-3 foot diameter typically); those boulders were usually made by dynamiting the original huge glacial boulders. There was one of the giant glacial boulders right on my street up until quite recently (the 1950's or something like that; the old neighbor was alive when it was still there); it's a shame that more weren't left and used as interesting city landscape features; of course it's an even bigger shame that more stands of old growth forest weren't left, we could have had stretches like the Golden Gate Park Panhandle running all over the city, and Seattle would then have been a unique and gorgeous city instead of the architectural garbage heap that it is today).

I've realized after buying this house we're in now that when shopping for a house, if you want to be a gardener, it's actually a liability to buy a house with a nice existing garden. The problem is you will want to work with what's already there, to respect those plants, and to save yourself work. But most likely the previous owner did some dumb things, planting big trees in bad places, or picking bad species or whatever. It's hard for me to just pull the trigger and rip out a garden that's already pretty nice, but if I had a crap garden I could plan it from the beginning and get more of what I want.

Typical "American Vernacular" style, taken from a real estate listing in my neighborhood :

Contrast with a Seattle park (Llandover Woods) that has very minimal sculpting, but is sticking to what this area should look like :

11-13-12 - Another Oodle Rewrite Note

Of course this is not going to happen. But in the imaginary world in which I rewrite from scratch :

I've got a million (actually several hundred) APIs that start an Async op. All of those APIs take a bunch of standard arguments that they all share, so they all look like :

OodleHandle Oodle_Read_Async(

                // function-specific args :
                OodleIOQFile file,void * memory,SINTa size,S64 position,

                // standard args on every _Async function :
                OodleHandleAutoDelete autoDelete,OodlePriority priority,const OodleHandle * dependencies,S32 numDependencies);

The idea was that you pass in everything needed to start the op, and when it's returned you get a fully valid Handle which is enqueued to run.

What I should have done was make all the little _Async functions create an incomplete handle, and then have a standard function to start it. Something like :

// prep an async handle but don't start it :
OodleHandleStaging Oodle_Make_Read(
                OodleIOQFile file,void * memory,SINTa size,S64 position

// standard function to run any op :
OodleHandle Oodle_Start( OodleHandleStaging handle,
                OodleHandleAutoDelete autoDelete,OodlePriority priority,const OodleHandle * dependencies,S32 numDependencies);

it would remove a ton of boiler-plate out of all my functions, and make it a lot easier to add more standard args, or have different ways of firing off handles. It would also allow things like creating a bunch of "Staging" handles that aren't enqueued yet, and then firing them off all at once, or even just holding them un-fired for a while, etc.

It's sort of ugly to make clients call two functions to run an async op, but you can always get client code that looks just like the old way by doing :

OodleHandle Oodle_Start( Oodle_Make_Read( OodleIOQFile file,void * memory,SINTa size,S64 position ) ,
                OodleHandleAutoDelete autoDelete,OodlePriority priority,const OodleHandle * dependencies,S32 numDependencies);

and I could easily make macros that make that look like one function call.

Having that interval of a partially-constructed op would also let me add more attributes that you could set on the Staging handle before firing it off.

(for example : when I was testing compresses on enwik, some of the tasks could allocate something like 256MB each; it occurred to me that a robust task system should understand limitting the number of tasks that run at the same time if their usage of some resource exceeds the max. eg. for memory usage, if you know you have 2 GB free, don't run more than 8 of those 256 MB tasks at once, but you could run other non-memory-hungry tasks during that time. (I guess the general way to do that would be to make task groups and assign tasks to groups and then limit the number from a certain group that can run simultaneously))


11-12-12 - The Destruction of American Democracy

People who actually care about democracy in America should be very concerned about the trend in the last 10 years. (granted if we look back farther to the LBJ era and before, corruption was rife in American politics, but it seems like it got better for a while, and now it's getting worse again).

1. Corporate Speech / Unlimited political spending.

Duh. Can we impeach Thomas and Scalia already? Every lawyer and judge in this country knows that they have no business being on the bench; they are literally the punch-line of law school jokes. It's a farce that we have such incompetent, corrupt, lazy, biased buffoons making some of the most important decisions in the country. (in case you aren't aware : in Citizen's United (as with countless other cases), Thomas and Scalia were known to be meeting with the supporters of the conservative side of the case).

Without campaign finance reform, there is no democracy. Both parties are just the parties of big corporations now. Both parties are the puppets of wall street, the military, the health-care complex, the cable companies, etc, and none of those interests will ever be harmed, no matter how much they fuck over the populace. Politicians are the puppets of money, and money wins elections. With the current court, the only way we'll get serious campaign finance reform would be with an ammendment, which is pretty unlikely in this day.

(in amusing absurdism, lots of states are going after political speech by labor unions, and the courts have so far been upholding it. While I basically agree that Unions should not be making political speech, or at least their members should be allowed to opt out of funding it, it's in odd opposition to allowing unlimited corporate political speech)

2. Electoral College.

I think everyone with a brain realizes now that the electoral college is a huge disaster that's ruining national politics. National elections hinge entirely on the results in a few swing states, thus national party platforms and attention are directed at the interests of those states. The majority of the country has almost no say in national elections. It's completely retarded.

The electoral college also means that the national elections are disporportionally controlled by the state governments of a few swing states, which gives those state governments massive power over the nation that we all must be concerned about.

3. Voter roll tampering.

Voter roll tampering should be really shocking to anyone of either party that respects the right to vote. At the moment it appears to be the Republicans who are mainly using this tool (certainly in the olden corrupt days, the dems were masters of it).

For a while the main tool was expunging criminals from the rolls (with collateral damage to non-criminals). The new tool of choice is "voter id" laws. Voter id sounds okay in theory, but in practice the point of the voter id laws is to remove some poor people and some old people from the rolls, because they tend to vote more Dem. I've heard some Dems say it's not a big deal, it's only 20,000 people or so that lose their right to vote, but of course that number is *massive* in the swing states.

In a wider view, the problem is that the party which gets into majority in the state government has the power to change the rules to affect future elections. That should raise some serious eyebrows, and brings us to the next point :

4. Gerrymandering.

It's completely ridiculous that the party in charge (in many states) gets to draw the voting districts.

I think a lot of people don't realize how powerful this is, or how widespread, or how ridiculous many of the districts are. (see for example : amusing maps and disturbing control ). We're being robbed right in front of our eyes and we're not doing anything about it, and they're laughing all the way to the bank; it's sickening.

If you win control of a state by even a tiny margin like 51-49 , you can rig the districts to go for your party by a huge margin, like 12-4. The way you do it is you put all the opposition support into a few districts that they will win in landslides, like 95-5, and then you spread out your support just thin enough to guarantee lots of wins, like 55-45 wins. (if you have a 51% majority of the state's population, you can split your state into just 2 districts where the opposition wins 95-5 , and 23 districts where you win 55-45, giving you a 23-2 majority from 51-49).

I've seen proposals that there should be better non-political committees to draw the voting districts (something like direct-elected long term seats), but I think they're all doomed to be corrupt eventually. I would much rather see the elimination of voting districts entirely, and instead use direct state-wide election (something like : you vote for your top 5 people and the people with the most votes get the seats). (multi-vote systems are also a big win for other reasons; they give non-mainstream candidates a better chance of winning, and allow viable 3rd parties to form)

(I understand that the idea of local districts is that you have a rep in your area to help address your local issues, but I think that's basically a myth; the only thing local representation does is encourage corruption, as the rep tries to get tax cuts and earmarks for business interests in their district)

amusing disclosure : my first ever software job was working on gerrymandering software ("redistricting software" but of course we all knew what it was for). It was a CAD package that had all the census data, and you could move the borders around and see the political balance in each district so that you could easily adjust the lines to get the voter ratios you wanted. We sold it to Cook County, which is one of the classic Dem-side gerrymanders (what they do is take little slices of Democratic Chicago and put them into the suburban districts that might otherwise go Republican).


11-09-12 - Rotten Systems

1. Excessively long and hidden contracts are the not-much-discussed open secret behind the destruction of consumer rights.

Anytime you do anything in America these days you are handed a fifty page contract (*) with all kinds of crazy clauses that nobody reads. And even if you do read it and object, what are you supposed to do? Not sign it? The competition has the exact same kind of abusive contract. As a consumer you can't choose to avoid them.

(* = if you are actually handed the real contract, that's rare and you should consider that company to be upstanding. In reality there is usually a clause somewhere that says "the full contract is available by request" and there's another 200 pages you don't know about that have even more exclusions. And even if you do request the full contract and actually get it, by the time you get it they've changed it and you no longer have the latest. And even if you like the terms you see and sign it, they'll change it the next day, and one of the clauses was "the agreement is superceded by changes to the contract" or some such. You may as well just sign a blank page that says "you may rape me however you choose" because they can always change the rules)

Tricky contracts are one of the forgotten evils behind the whole mortgage crisis. Asshole type-A republican types will say "it's your own fault if you don't read the fine print; you were given a contract and agreed to it, you have to live with it". Bullshit. We are all presented with reams of intentionally-obfuscated lawyer-speak, it is totally unreasonable to allow it.

There need to be laws about limitting the complexity of contracts. There need to be laws about limitting the amount of fine print in fee structures; pricing needs to be up front and standardized and clearly advertised.

Things like the classic "It's only $9.95 (with ten more monthly payments of 99 thousand)" should just be illegal, obviously. There's no social benefit in allowing that kind of advertising. The point of laws is to make a capitalist/social structure which is good for the people, and that kind of shit is not good; it's particularly bad for capitalists who want to play fair by selling a good product with an honest price.

The new evil in abusive contracts is of course software license agreements. These should just be completely illegal. You shouldn't be allowed to compel me to sign anything in order to use the product that I already bought. The purchase implies a standard obligation of functionality and indemnity, that should be the only agreement allowed. The standard indemnity laws should be updated a bit to reflect the modern age of software of course.

2. America desperately needs a real right-to-privacy law.

2.A. Government agencies should not be allowed to sell your personal information to corporations! (among others, the USPS does this, as do most state DMV's). This one is a super no-brainer.

2.B. No corporation should be allowed to sell your personal information without your explicit permission. But really what's really needed is :

2.C. Corporations should not be allowed to require any personal information beyond some bare minimum that can identify you. eg. they can ask for SSN and a password of your choosing, but they cannot ask for previous addresses or bank account numbers or etc.

2.D. It should be illegal to tie incentives to privacy violations. eg. club cards that give you discounts in exchange for giving up your privacy. Similarly lots of bills now give you a discount if you allow direct withdrawal from your bank account. Utilities often will allow you to not give your SSN, but only with a fee.

2.E. All privacy options must be set to max-privacy by default. Allowing increased privacy but requiring you to go through forms is no good.

2.F. You should be able to request deletion of all your records. eg. when you close a bank account, or from a doctor, or whatever, you should be able to say "please delete all your info on me" and they should be required by law to comply (if your accounts are in good standing blah blah blah).

3. I feel like the government-corporate complex is intentionally building this world structure in which you are locked into a variety of fixed fees which suck up all your income. (okay this part of the post is going off the deep end a bit)

You don't go work and get your money and then choose how to spend it. The corporate masters have auto-debit on your account and just suck it right out. I know this is retarded hyperbole, but it sort of feels like mining towns where you get paid scrip and then just have to give it right back at the company store, but in this case the company store is apple and the chinese-crap importers (target, walmart, gap, c&b, etc etc).

3.1. Health insurance is perhaps the most obvious; the cost of health care is ridiculously inflated, but that cost is hidden from you a bit (intentionally), so we're all just locked into paying out a massive amount monthly to the health care complex.

3.2. Car insurance is of course the same story. The car insurance companies very much want you to feel like "accidents are no big deal" ; hey lets all jump in our tanks with no visibility and smash into each other, no biggie, the car insurance pays for it. And in the mean time a huge government-required deduction slips out of your account every month.

3.3. Cell phones obviously; and Cable companies; these ones are semi-government-enforced monopolies, and basically not optional in modern life, you are just required by law to give them $200 every month. More and more software wants to move to subscription plans. Everyone is getting very clever about making the easy way out just being "give us lots of money every month automatically" , and you have to work harder and harder to actually be proactive about spending your money.

3.4. I actually think online purchasing is part of this and is changing the whole relationship people have with money. You very often no longer get to see the thing you are buying before you buy it. Then when it sucks, it's usually too much trouble to return it. The result is that the whole purchasing is more like "send money out into the ether and then products show up which I have no control over". It's almost like a constant tax, and then they give you some shitty products once in a while.


11-08-12 - Job System Task Types

I think I've written this before, but a bit more verbosely :

It's useful to have at least 3 classes of task in your task system :

1. "Normal" tasks that want to run on one of the worker threads. These take some amount of time, you don't want them to block other threads, they might have dependencies and create other jobs.

2. "IO" tasks. This is CPU work, but the main thing it does is wait on an IO and perhaps spawn another one. For example something like copying a file through a double-buffer is basically just wait on an IO op then do a tiny bit of math, then start another IO op. This should be run on the IO thread, because it avoids all thread switching and thread communication as it mediates the IO jobs.

(of course the same applies to other subsytems if you have them)

3. "Tiny" / "Run anywhere" tasks. These are tasks that take very little CPU time and should not be enqueued onto worker threads, because the time to wake up a thread for them dwarfs the time to just run them.

The only reason you would run these as async tasks at all is because they depend on something. Generally this is something trivial like "set a bool after this other job is done".

These tasks should be run immediately when they become ready-to-run on whatever thread made them rtr. So they might run on the IO thread, or a worker thread, or the main thread (any client thread). eg. if a tiny task is enqueued on the main thread and its dependencies are already done, it should just be run immediately.

It's possible that class #2 could be merged into class #3. That is, eliminate the IO-tasks (or GPU-tasks or whatever) and just call them all "tiny tasks". You might lose a tiny bit of efficiency from that, but the simplicity of having only two classes of tasks is probably preferable. If the IO tasks are made into just generic tiny tasks, then it's important that the IO thread be able to execute tiny tasks from the generic job system itself, otherwise it might go to sleep thinking there is no IO to be done, when a pending tiny IO task could create new IO work for it.


Beyond that, for "normal" tasks there's the question of typical duration, which tells you whether it's worth it to fire up more threads.

eg. say you shoot 10 tasks at your thread-pool worker system. Should you wake up 1 thread and let it do all 10 ? Or wake up 10 threads and give each one a task? Or maybe wake 2?

One issue that still is bugging me is when you have a worker thread, and in doing some work it makes some more tasks ready-to-run. Should it fire up new worker threads to take those tasks, or should it just finish its task and then do them itself? You need two pieces of information : 1. are the new tasks significant enough to warrant a new thread? and 2. how close to the end of my current task am I? (eg. if I'm in the middle of some big work I might want to fire up a new thread even though the new RTR tasks are tiny).

When you have "tiny" and "normal" tasks at the same priority level, it's probably worth running all the tinies before any normals.

Good lord.


11-06-12 - Using your own malloc with operator new

In Oodle, I use my own allocator, and I wish to be able to still construct & destruct classes. (Oodle's public API is C only, but I use C++ internally).

The traditional way to do this is to write your own "operator new" implementation which will link in place of the library implementation. This way sucks for various reasons. The important one to me is that it changes all the news of any other statically-linked code, which is just not an okay thing for a library to do. You may want to have different mallocs for different purposes; the whole idea of a single global allocator is kind of broken in the modern world.

(the presence of global state in the C standard lib is part of what makes C code so hard to share. The entire C stdlib should be a passed-in vtable argument. Perhaps more on this in a later post.)

Anyway, what I want is a way to do a "new" without interfering with client code or other libraries. It's relatively straightforward (*), but there are a few little details that took me a while to get right, so here they are.

(* = ADDENDUM = not straightforward at all if multiple-inheritance is used and deletion can be done on arbitrary parts of the MI class)



subtlety : just calling placement new can be problematic; it's safer to make an explicit selection
of placement new.  This is how we call the constructor.


enum EnumForPlacementNew { ePlacementNew };

// explicit access to placement new when there's ambiguity :
//  if there are *any* custom overrides to new() then placement new becomes ambiguous   
inline void* operator new   (size_t, EnumForPlacementNew, void* pReturn) { return pReturn; }
inline void  operator delete(void*,  EnumForPlacementNew, void*) { }

// subtlety : some stdlibs have a non-standard operator new with alignment (second arg is alignment)
//  note that the alignment is not getting passed to our malloc here, so you must ensure you are
//    getting it in some other way
inline void* operator new   (size_t , size_t, EnumForPlacementNew, void* pReturn) { return pReturn; }

// "MyNew" macro is how you do construction


subtlety : trailing the arg list off the macro means we don't need to do this kind of nonsense :

    template <typename Entry,typename t_arg1,typename t_arg2,typename t_arg3,typename t_arg4,typename t_arg5,typename t_arg6,typename t_arg7,typename t_arg8,typename t_arg9>
    static inline Entry * construct(Entry * pEntry, t_arg1 arg1, t_arg2 arg2, t_arg3 arg3, t_arg4 arg4, t_arg5 arg5, t_arg6 arg6, t_arg7 arg7, t_arg8 arg8, t_arg9 arg9)


//  Stuff * ptr = MyNew(Stuff) (constructor args); 
//  eg. for void args :
//  Stuff * ptr = MyNew(Stuff) ();
#define MyNew(t_type)   new (ePlacementNew, (t_type *) MyMalloc(sizeof(t_type)) ) t_type 

// call the destructor :
template <typename t_type>
static inline t_type * destruct(t_type * ptr)
    RR_ASSERT( ptr != NULL );
    return ptr;

// MyDelete is how you kill a class


subtlety : I like to use a Free() which takes the size of the object.  This is a big optimization
for the allocator in some cases (or lets you not store the memory size in a header of the allocation).
*But* if you do this, you must ensure that you don't use sizeof() if the object is polymorphic.
Here I use MSVC's nice __has_virtual_destructor() extension to detect if a type is polymorphic.


template <typename t_type>
void MyDeleteNonVirtual(t_type * ptr)
    RR_ASSERT( ptr != NULL );
    #ifdef _MSC_VER
    RR_COMPILER_ASSERT( ! __has_virtual_destructor(t_type) );
    MyFree_Sized((void *)ptr,sizeof(t_type));

template <typename t_type>
void MyDeleteVirtual(t_type * ptr)
    RR_ASSERT( ptr != NULL );
    // can't use size :
    MyFree_NoSize((void *)ptr);

#ifdef _MSC_VER

// on MSVC , MyDelete can select the right call at compile time

template <typename t_type>
void MyDelete(t_type * ptr)
    if ( __has_virtual_destructor(t_type) )


// must be safe and use the polymorphic call :

#define MyDelete    MyDeleteVirtual


and the end result is that you can do :

    foo * f = MyNew(foo) ();


and you get normal construction and destruction but with your own allocator, and without polluting (or depending on) the global linker space. Yay.

11-06-12 - Protect Ad-Hoc Multi-Threading

Code with un-protected multi-threading is bad code just waiting to be a nightmare of a bug.

"ad-hoc" multi-threading refers to sharing of data across threads without an explicit sharing mechanism (such as a queue or a mutex). There's nothing wrong per-se with ad-hoc multi-threading, but too often people use it as an excuse for "comment moderated handoff" which is no good.

The point of this post is : protect your threading! Use name-changes and protection classes to make access lifetimes very explicit and compiler (or at least assert) moderated rather than comment-moderated.

Let's look at some examples to be super clear. Ad-Hoc multi-threading is something like this :

int shared;

thread0 :

shared = 7; // no atomics or protection or anything

// shared is now set up

start thread1;

// .. do other stuff ..

kill thread1;
wait thread1;

print shared;


thread1 :

shared ++;


this code works (assuming that thread creation and waiting has some kind of memory barrier in it, which it usually does), but the hand-offs and synchronization are all ad-hoc and "comment moderated". This is terrible code.

I believe that even with something like a mutex, you should make the protection compiler-enforced, not comment-enforced.

Comment-enforced mutex protection is something like :

struct MyStruct s_data;
Mutex s_data_mutex;
// lock s_data_mutex before touching s_data

That's okay, but comment-enforced code is always brittle and bug-prone. Better is something like :

struct MyStruct s_data_needs_mutex;
Mutex s_data_mutex;
#define MYSTRUCT_SCOPE(name)    MUTEX_IN_SCOPE(s_data_mutex); MyStruct & name = s_data_needs_mutex;

assuming you have some kind of mutex-scoper class and macro. This makes it impossible to accidentally touch the protected stuff outside of a lock.

Even cleaner is to make a lock-scoper class that un-hides the data for you. Something like :


template <typename t_data> class ThinLockProtectedHolder;

template <typename t_data>
class ThinLockProtected

    ThinLockProtected() : m_lock(0), m_data() { }
    ~ThinLockProtected() { }    


    friend class ThinLockProtectedHolder<t_data>;
    OodleThinLock   m_lock;
    t_data          m_data;

template <typename t_data>
class ThinLockProtectedHolder

    typedef ThinLockProtected<t_data>   t_protected;
    ThinLockProtectedHolder(t_protected * ptr) : m_protected(ptr) { OodleThinLock_Lock(&(m_protected->m_lock)); }
    ~ThinLockProtectedHolder() { OodleThinLock_Unlock(&(m_protected->m_lock)); }
    t_data & Data() { return m_protected->m_data; }

    t_protected *   m_protected;


#define TLP_SCOPE(t_data,ptr,data)  ThinLockProtectedHolder<t_data> RR_STRING_JOIN(tlph,data) (ptr); t_data & data = RR_STRING_JOIN(tlph,data).Data();


// use like :

    ThinLockProtected<int>  tlpi;
    shared_int = 7;



So the point of this whole post is that even when you are just doing ad-hoc thread ownership, you should still use a robustness mechanism like this. For example by direct analogy you could use something like :


template <typename t_data> class AdHocProtectedHolder;

template <typename t_data>
class AdHocProtected

    AdHocProtected() : 
    #ifdef RR_DO_ASSERTS
        m_data() { }
    ~AdHocProtected() { }


    friend class AdHocProtectedHolder<t_data>;
    #ifdef RR_DO_ASSERTS
    U32             m_lock;
    t_data          m_data;

void AdHoc_Lock( U32 * pb)  { U32 old = rrAtomicAddExchange32(pb,1); RR_ASSERT( old == 0 ); }
void AdHoc_Unlock(U32 * pb) { U32 old = rrAtomicAddExchange32(pb,-1); RR_ASSERT( old == 1 ); }
#define AdHoc_Lock(xx)
#define AdHoc_Unlock(xx)

template <typename t_data>
class AdHocProtectedHolder

    typedef AdHocProtected<t_data>  t_protected;
    AdHocProtectedHolder(t_protected * ptr) : m_protected(ptr) { AdHoc_Lock(&(m_protected->m_lock)); }
    ~AdHocProtectedHolder() { AdHoc_Unlock(&(m_protected->m_lock)); }
    t_data & Data() { return m_protected->m_data; }

    t_protected *   m_protected;


#define ADHOC_SCOPE(t_data,ptr,data)    AdHocProtectedHolder<t_data> RR_STRING_JOIN(tlph,data) (ptr); t_data & data = RR_STRING_JOIN(tlph,data).Data();

which provides scoped checked ownership of variable hand-offs without any explicit mutex.

We can now revisit our original example :

AdHocProtected<int> ahp_shared;

thread0 :


shared = 7; // no atomics or protection or anything

// shared is now set up

start thread1;

// .. do other stuff ..

kill thread1;
wait thread1;

print shared;


thread1 :

shared ++;


And now we have code which is efficient, robust, and safe from accidents.

old rants