Thursday, May 28, 2015

Python: Implicit is better than explicit

Why the slogan?


I'm a big fan of Python. The syntax is great; the open-source community around it is extremely helpful; and the language lets me focus on the problems I'm trying to solve rather than the implementation details of the code.

A very peculiar feature of Python is that it comes with a set of ethical or aesthetic guidelines. My favorite -- because it's the most perplexing -- is the motto: "Explicit is better than implicit". It's perplexing because although it seems really simple (and maybe even obvious) when you read it, the more you think about it, the less clear it becomes. In fact, I think it's at least misleading, and probably wrong.

Like any programming language, Python makes a number of compromises. It aims for simplicity, and therefore hides quite a lot of details concerning how the interpreter works. Of course, you give up a lot when you don't understand the inner workings of the language. But Python allows you to surface a lot of its inner machinery if you want it. So you have the choice between either cruising along, happily skimming along the surface of the language and taking a deep dive into it.

One of the most conspicuous items that's missing from Python is type-checking. Functions don't care what they're passed, and no checking occurs before run-time. Instead, functions heroically try to perform whatever operations they've been told to perform, regardless of the types of their arguments. If the operation can be carried out, it's done. If it can't, we throw an exception.  This is known, of course, as "duck-typing". Python doesn't care whether the thing actually is a duck. So long as it behaves like a duck, we'll call it a duck.

Here's an easy example. Suppose you've written a function that adds two things together.


1
2
def my_function(x, y):
    return x + y

We haven't had to declare the types of our function's arguments. It'll accept anything, and blithely try to add the arguments together. Here's where the duck-typing comes in. If we pass two integers to the function, it'll behave as expected by returning the sum. But if we pass in two strings, the function will also work: namely, by concatenating the strings together.

Python doesn't bother checking the types of x and y. Instead, it looks up their types and tries to find a method corresponding to the "plus" sign. If it finds one, the method is applied. So the function will work for strings, lists, ints, and floats, but not for sets. The upshot is that if you see a snippet of code that uses the plus sign, you have no idea whatsoever what that code does. The meaning of the plus sign is not explicit -- it's implicit.

It easy to come up with other examples in which the value of implicit functionality has been baked into Python. For instance, a common "gotcha" for people who are new to Python is how variables are passed to functions. Sometimes, they're passed by value. Other times, they're passed by reference. Mutable objects are passed by reference; immutable objects by value. Nothing in the code indicates this. It's implicit in the object's type. To take another example, method resolution order for multiple inheritance has a lot of implicit functionality under the hood. Which methods are used by a class with multiple inheritance is not something that's explicitly stated anywhere in the code.

Other than purely declarative programming languages, Python may have more implicit functionality than any other language I've come across. Which is why the "Explicit is better than implicit" slogan is so puzzling, and why it's the source of so many rants and arguments on StackOverflow.

I believe in *magic


The "explicit is better than implicit" slogan probably explains why there's such widespread resistance to so-called "magic" in Python. If you ask a question on StackOverflow about how to implement some Python magic -- for example, how to dynamically create classes or functions, or how to overload Python's import mechanisms -- you'll definitely get at least one answer saying, "Don't do it!". Python magic increases the amount of implicit functionality, and so there's often an assumption that if there's any other way to get something accomplished, magic should be avoided. So the fact that Python makes magic so easy is also another source of tension when people argue about the "best" way to implement something.

Personally, I think there's nothing at all wrong with having lots of magic and lots of implicit functionality in your Python code, even if it's not absolutely necessary. In fact, I think it's great! A couple of examples will help explain why I'm so strongly in favor of Python magic.

For my own work as a software engineer, I often write modules that are intended to be useful to other engineers. But of course, the mere fact that a piece of code would be useful doesn't by itself get other people to use it. There are high barriers to entry when you're learning how to use a new module. A common feature of modules that have a steep learning curve is that they require a pattern used in one piece of code to line up in some complicated with another pattern used in another piece of code. Consider a really simple example. Suppose you've got two functions that look like this:

def sender(some_dictionary):
    do_something(...)
    new_dictionary = {
        k, v for k, v in some_dictionary.iteritems()
        if some_test(v)}
    receiver(**new_dictionary)

def receiver(foo, bar):
    do_something(foo, bar)

The two functions have to "line" up correctly in order to avoid throwing an exception. The "sender" function gets a dictionary and builds a new dictionary by extracting some subset of its key-value pairs. The new dictionary is sent to "receiver" as a set of **kwargs.

If I were writing these functions for myself, I wouldn't bother doing anything fancy. But if I intended for someone else to be able to quickly implement functions like this, I'd worry a little bit. I'm forcing the other person to keep track of exactly which components of the dictionary are relevant to the "receiver" function. And this creates a set of "gotchas" that can be the source of much frustration. To take one example, this can happen if the module requires the user to subclass and provide a couple of methods that are required by the base class. (Scrapy works like this, for instance.)

Instead, why not let the code inspect the "receiver" function, get a list of its arguments, and automagically pare down the dictionary to include only the keys that the function needs? Python makes this easy to accomplish. The new dictionary can be programmatically defined like this:

{k: v for k, v in some_dictionary.iteritems()
 if k in if k in inspect.getargspec(receiver).args}

This functionality could be baked into some base classes, or put into a decorator. It's magic, but so what? It would be nutty to write something like this if there was no intention for others to use the relevant code. But at the cost of a small bit of complexity and a bit of magic, we can free other people from having to worry about how the "sender" and "receiver" functions align with each other.

"Simple", "Complex", and "Complicated" don't mean anything


The "Explicit is better than implicit" slogan seems really plausible because it seems to be implied by the principles that "Simple is better than complex" and "complex is better than complicated". When a Python script does something implicitly, the code frequently becomes more complex. For a really clear example of this, check out the "sh" module, which hijacks Python's import mechanism and uses it to create functions that execute external commands. The effect of this is that you can "from sh import foo" and get back a "foo" function, even though no such function is actually defined in the sh module.

The module does this by doing a number of clever tricks. But is the module "complex? Or worse yet, "complicated?" There's no answer. Or more precisely, the answer depends on who you are. If you're a user of the module, there's nothing simpler. If you're a casual programmer poking around in the module's code, or someone who's looking to modify it, then it's probably "complex" (at least).

Simplicity isn't a characteristic of anything. It's a relationship. In programming, it's a relationship between three things: the code, the user of the code, and the purpose to which the code is being put. When those three align in the right way, you've got simple (and elegant) code. If they're out of alignment, then you've got something else -- in the worst case, it's complicated. The same code can be both simple and complex.

And this is why we too often assume that explicit is better than implicit. In typical use-cases, implicit creates complexity because of who the users are and what they're using the module for. But in many contexts, the opposite is true.

Monday, May 25, 2015

Be an ethical engineer

I made a career transition from academia to software engineering about a year and a half ago. It's been a very good move for me, for many reasons. If you're an intellectually curious person and you're comfortable making a big change, there's nothing better than switching careers. You'll be on an almost vertical learning curve; you'll learn stuff you didn't know existed; and you'll meet a lot of people you wouldn't have met before.

One of the many things I've learned is just how hard we're pushing the boundaries of existing technology. I'm sure the vast majority of people don't have any idea how much effort, time, money, and brainpower is required to build the computer systems we use every single day. I won't get into details here, but if you use any popular web-based service like Google, Netflix, Facebook, or Twitter, you're relying on a vast infrastructure comprising thousands upon thousands of computers spread out all over the world. Data centers are so large that they have to be built in specific locations where there's enough electricity available; and they are often built near large sources of water so that the water can be pumped through the facility and used for cooling. Even small tech start-ups with just a few employees can easily require hundreds of machines networked together in extremely complex ways.

This is why engineers are so valuable to industry and government. It is not easy to get these systems up and running, and it requires a skill set that's not easy to find. At the company where I work, we have been hiring engineers constantly for years. There hasn't been a time when we weren't interviewing people, spending money on recruiters, going to job fairs, or doing something else as part of the never-ending search for skilled engineers. The entire tech industry is in an escalating competition for talented people. Every technical talk I attend either begins or ends with a plea for people to apply for jobs at their respective companies.

Getting your hands on huge amounts of computing power is quick, simple, and cheap. But raw computing power is worthless by itself. Transforming those resources into something useful requires engineering know-how. And the demand for that knowledge is outstripping supply by a very wide margin. This gives engineers the rare luxury of choice among potential employers. The engineers I've met value tackling intellectually challenging problems, learning about new technologies, the autonomy to make their own creative decisions, the company of like-minded people, a collaborative environment, and the opportunity to see their work make a significant impact on the world. And they typically don't need to make very many compromises when they're looking for a job.

For all these reasons, we've got a major opportunity to make the world better by taking our ethical values into account when we're looking for work. Take, for example, the National Security Agency's (NSA) new data center. Here's how The Economist describes it:
Deep in the desert in Utah, the National Security Agency... has built a $1.5 billion centre to scoop up and analyse data from the internet. The building includes its own water-treatment facility, electric substation, and 60 back-up diesel generators. It will use over a million gallons of water a day. Its data-storage capacity would be enough, according to one estimate, to store a year of footage of round-the-clock video-recording of over a million people. At this centre, communications from across the globe are tapped directly from the fibre-optic backbone of the internet.
I happen to think that the surveillance directives handed down to the NSA are unethical. You may agree or disagree. But it's certainly true that a facility like the one in Utah requires a large group of engineers with highly specialized and rare skills. It takes a lot of really smart people to design, build, and maintain a behemoth like this one. The number of people capable of handling an engineering challenge like this one is very, very small. And that tiny group of engineers makes it possible for such facilities to exist.

Without those people, the NSA facility would go up in flames. Literally. It's easy to come up with other examples of complex facilities and technologies that are morally suspect. In my opinion, high-speed trading on Wall Street is enormously unethical because it creates volatility, concentrates wealth artificially, and distracts from the real purpose of a market -- namely, to enable capital to flow from unproductive to productive uses. Here again, there's a rare set of skills that makes this activity possible.  An engineer with the ability to shave a couple of nanoseconds off the time required for a signal to propagate through a network is potentially worth vast sums of money. But there are very few people capable of doing this. High-speed trading would collapse if it lacked the people with the know-how to build and maintain these systems.

Furthermore, there's a huge difference productivity between a top-notch "A-Player" engineer and a merely competent "B-Player" engineer. Employers take this very seriously. Smart employers will spend huge amounts of time and money seeking out "A-Players" to hire, and they have no qualms about leaving a vacancy open for a very long time in order to avoid hiring someone who is merely competent. For this reason, a highly skilled engineer deals a major blow to any organization he or she refuses to work for, even if someone eventually fills the position. For all these reasons, engineers have the means, motive, and opportunity to improve the world. You simply have to take into account your own values when deciding who to work for.

Sunday, May 17, 2015

The "Follow Your Passion" Trap

I just read an interesting post in the "what's the matter with the youth today?" genre. It's about how the youth today don't struggle and therefore never develop a passion for anything, because passion requires struggle and lots of hard work. Like a lot of observations in this genre, there's probably some truth to it. But I think it really misses the important reasons why people lead lives filled with mind-numbing work they have no passion for.

First, an important disclaimer. Obviously, a huge number of people spend their lives performing thankless, unfulfilling, rote work simply because they have no other choice. The reason why these people might not be able to name something they're passionate about is because they're exhausted. So when we're talking about people who aren't passionate about anything, we're really discussing a problem that only lucky people have. We're talking about people who have the luxury of time and resources -- people who ought to be able to follow their passions, but don't, for the simple reason that they're not passionate about anything.

Second, it would be nice to get clear on what a "passion" is. Unfortunately, we can't. But we can tell when someone is passionate about something, just as Supreme Court Justice Potter Stewart admitted that he couldn't define pornography, but said that he knows it when he sees it. There are certainly symptoms that a person displays when they are passionate about something. They spend a lot of time pursuing it; they care deeply about becoming skilled at it; they find it intrinsically worthwhile, and not just a means to an end. Often, they have a difficult time explaining to others why they care so deeply about their passion. They often seem obsessive, and they get a lot of satisfaction by becoming more skilled at it, or by achieving goals associated with it. Conversely, when they can't pursue their passion, they become frustrated, angry, or depressed.

I give a lot of (solicited) advice to people about switching careers, because this is something I've done recently. Several of these people have expressed frustration, and have told me something like this:
You're lucky. You have something you're passionate about, so it was easy for you to decide what career you wanted to move into. But I don't. There isn't anything I care deeply about, so I don't know what to do!
This is a really pernicious attitude, because it requires an assumption that's utterly self-defeating, but is difficult to expose. We'll have to approach this assumption indirectly, and sneak up on it.

Let's not think about being passionate about work, but about being in love with a person. It's not a bad analogy; after all, many of symptoms of being passionate resemble the symptoms of being in love. Consider a person who we'll call "Bob". Bob isn't in love with anyone, but wants to be. Bob complains one day:
Look at those happy couples. They're so lucky. It would be good to be in love, but I'm not in love with anyone. I'm just not an in-love kind of person. If I were that kind of person, then I'd be in love with someone.
Bob is kind of nutty. There isn't an "in-love" kind of person who's in love and therefore does things with the object of their love. Bob is not only nutty, but also fatalistic. If he really thinks that he's just not that kind of person, then Bob is very unlikely to ever fall in love with anyone.

If you were Bob's friend, you'd probably want to slap him and say:
"Bob, it doesn't work like that! You don't start out being in love. You meet people and spend time with them, and then maybe you fall in love with one of them. You don't spend time with someone because you're in love -- the feeling of being in love comes later!"
Now let's get back to the problem of having a passion. When we say to someone, "follow your passion", we're setting up a subtle trap. We're suggesting that people have passions first, and then they pursue those passions second. It's like telling Bob to spend time with the person he loves. Not only would this be unhelpful advice, but it would also be highly demoralizing.

So what I tell people is this: Of course you don't have a passion -- you haven't tried anything! For most of us, we don't switch careers or make any other kind of big change because we already know what our passion is. We have to just take a stab at something, and try to keep an open mind about discovering something we can learn to care deeply about. This is a very scary proposition. It will probably turn out that you try something that doesn't work out -- in fact, this is overwhelmingly likely. This may feel like a failure, and so you'll probably feel that you've failed several times at least. You will doubt yourself, and your self-esteem will take some very big hits. Many of the lessons you learn will be negative -- mistakes that you promise to yourself you won't make again. But unfortunately, we are not born with our own personal set of passions that guide us through our lives. Passions are discovered by trial-and-error, and then developed over time. They do not guide you. Instead, you have to wander through the dark while struggling to keep an open mind. And if you're lucky, you'll find the right opportunity. Then the real work begins.

Saturday, May 2, 2015

Shut Down Your PhD Program

When I was a graduate student in an excellent philosophy department, there was a conventional wisdom about the academic job market. Generally, people believed that although the market was bad, it would soon improve dramatically as the large number of faculty hired during the late 1960s and 1970s retired. Their retirement would create a lot of job openings and put the academic job market back in balance.

That was in the late 90s and early 2000s. Since then, the academic job market has gotten steadily worse. There are plenty of reasons for this, some of which we couldn't predict at the time. As the economy got worse, senior faculty delayed their retirements. State appropriations for colleges and universities declined, and administrators were eager to recover whatever money they could by either not filling faculty lines, or (more often) filling them with temporary or adjunct positions, which often have criminally low salaries.

When I see humanities PhDs on the job market, I'm horrified. I thought I had it tough when I went on the market in 2001. But that was a walk in the park compared to the current situation. People who could have landed excellent jobs fifteen years ago are out of luck today. I'm pretty sure that in my own case, although I happened to land a terrific job out of graduate school, I'd have a tough time today finding anything if I were doing it over again.

There is no indication whatsoever that the job market for highly qualified, hard-working PhDs is likely to improve anytime in the foreseeable future. Those retirements didn't have any measurable effect, the hiring of adjuncts and other temporary, non-tenure track faculty is only accelerating, and budgets are low. Although we're out of the recession in the United States, it is not like higher education has bounced back.

In this climate, academic departments with PhD programs are under an obligation to justify their existence -- this is especially true of departments whose graduate programs are of average quality or below. When I was a tenured faculty member in such a department, we justified the existence of our PhD program with something like this (although we wouldn't be so explicit about it): "We're fairly average now, but we are having good luck placing our PhDs into academic positions (relative to other, similar programs), and we're doing a good job training them. More importantly, we have a lot of administrative support for improving our program. We're able to hire additional tenure track faculty, and we've got a solid plan for becoming an excellent program, especially in a few strategically selected areas of specialization."

I have since left academia entirely, in part because I started to appreciate how delusional that justification was. Year after year, we collectively shot ourselves in the foot and squandered one opportunity after another. Our graduate students had an increasingly difficult time finding jobs, funding declined, and we became overwhelmed by the institutional inertia that characterizes so much of academia. Academic departments are very rarely capable of making the qualitative changes that are necessary in order to improve dramatically. It's crazy to think that the leadership that brought about the status quo and flourished in it will suddenly become motivated to institute major changes, or for that matter, that they'll be competent to do it. I don't think my department was unusual at all in this regard. This trajectory seems to me at least to be quite typical.

For these reasons, our justification for having a PhD program was flimsy at best. But I'm not too disturbed by that, mainly because our purported reasons were not our actual reasons. The fact is, faculty tend to enjoy working in a department with a PhD program. It's usually a bit more prestigious, the pay is somewhat better on average, there's less teaching and more freedom to do research. And of course, graduate students do a lot of the work that faculty don't want to do. There are downsides, of course. There's extra administrative work, and the time spent supervising graduate students is significant (although I found this very enjoyable). But on the whole, for most faculty it's quite desirable to have such a position. Faculty and administrators want to have a PhD program because of self-interest.

This would be alright if having a PhD program didn't do any harm. In other words, someone might try to justify their PhD program by saying something like this: "It's not as if we are damaging our graduate students by training them in our discipline; and after all, they are adults, and nobody forced them to go to graduate school or to choose our program. If it wasn't a benefit to them, they wouldn't have applied in the first place; and nobody is forcing them to stay, either."

If this is the justification for a PhD program, then it's important to understand what this argument really amounts to. It's an economic argument, pure and simple. It asserts that the market for one's PhD program demonstrates its utility, exactly like a market for widgets would be used by an economist to demonstrate the value of widgets. And because it is an economic argument, it's subject to the same criticisms we commonly level against that kind of reasoning.

Market-based arguments for the value of something assume that the buyers are informed. But are potential graduate students informed of what academia is really like, and more importantly, of what the academic job market is like? Personally, I'm skeptical. It's one thing to be told how bad it is, but it's very difficult to fully appreciate that the job market is a post-apocalyptic nightmare. I know I didn't (and I had it way better than today's graduate students). From my years with graduate students -- even very smart ones -- I don't think most of them understood, either. To be sure, the truth would gradually sink in, but only after a significant amount of time and money was invested.

Furthermore, arguments that appeal to a market also assume that buyers can switch if they don't like what they buy. And certainly, graduate students can switch by simply quitting. Proponents of PhD programs will no doubt say that there's nothing easier than waking up in the morning and not going to graduate school. In short, if you don't like it, then go do something else.

This is a line of reasoning I don't accept at all. When someone goes to graduate school, they dedicate a huge proportion of their time and effort to their studies. This means that they immerse themselves in the culture of their graduate program. However inadvertently, faculty inculcate in their graduate students the attitude that quitting graduate school is to be defeated. It is to be not as smart, dedicated, or hard-working as their fellow students who choose to stay. I think that as faculty, we underestimate how powerfully we send this message, and how psychologically difficult it is to overcome it. Leaving a graduate program is very difficult. And faculty bear some responsibility for this.

No doubt, some people will be offended by my paternalistic tone. If you're one of those people, then so be it. More interestingly, however, I think there's a good self-interested reason for faculty to be opposed to the existence of so many PhD programs. The argument is also market-based, but it doesn't appeal to dubious assumptions about market efficiency.

Every faculty member laments the fact that tenure-track jobs are being replaced by non-tenure track jobs. Adjuncts are typically exploited in all the familiar ways that I don't need to discuss here. Furthermore, the lack of new tenure-track positions weakens the bargaining power of faculty. The more reliant higher education becomes on temporary workers, the fewer tenured faculty there are, and the less influential they are as a whole. Although faculty often don't like to think of themselves as workers, it is nonetheless true that tenured faculty form a de facto union because they have similar interests and the extraordinary job security of tenure, which gives them an unusual amount of bargaining power (if they choose to use it).

Everyone knows that one way you break a union is by hiring lots of non-union workers. In this case, the non-union workers are the adjuncts and the graduate students. Tenured and tenure-track faculty who have PhD programs are effectively recruiting and training their own scabs! We don't think of graduate students and adjuncts as playing the same role in this dynamic, but there's no difference between them insofar as they weaken the bargaining power of faculty, reduce the number of tenure track jobs, and empower administrators to continue to make asinine strategic decisions on behalf of the university. Of course, with graduate students gone, the university could redouble its efforts to hire adjuncts. But even this would be better because it would highlight the short-term and destructive decisions that are being made by bloated and uninformed administrations.

For all of these reasons, the vast majority of PhD programs harm everyone in the university, and an important step toward slowing the decline of higher education in the United States is to shut them down.

Saturday, April 25, 2015

Living With Chronic Pain

Like millions of other people, I have chronic pain. In my case, it happens to be a chronic migraine condition called -- accurately enough -- "daily chronic migraine". Migraines count as "daily" if they occur at least fourteen times per month, although that's somewhat of an arbitrary number of course. In my case, my migraine is twenty-four hours per day, seven days a week. It does not take off for either weekends or holidays. This is more common than you might think. Millions of Americans have chronic migraine, and many of them have it, like I do, constantly.

The two most common questions I get from people about my chronic pain are: (1) Really? You have a migraine right now?, and (2) Have you tried...?. The answer to both questions is "yes". When I say that I have a migraine 24/7, this is not hyperbole; I mean this quite literally. This logically entails that I have a migraine right now. I also receive many very well-intentioned suggestions for treatment options. As much as I appreciate the thought behind these suggestions, I'm frankly no longer interested in hearing about them. I've tried them all. I have been on every over-the-counter medication and supplement (especially vitamin D and coenzyme Q10), I was prescribed no fewer than twenty-seven different medications, many of which I took in various combinations and in varying doses. I've been hospitalized several times for treatments that have to be conducted under constant observation because they're dangerous. I had Botox and another nerve block (whose name escapes me) injected into various nerves. At one point (at a doctor's suggestion), I started inhaling Lidocaine into my nose to abort these attacks. And yes, I have tried acupuncture. I've had every kind of test, including a huge number of blood tests, half a dozen MRIs, a CAT scan, and a cranial angiogram, for which a cable is inserted from your groin into your skull so that an image can be made from inside the skull. I've even had a brain biopsy, in which a small piece of brain tissue is removed and sliced into a zillion tiny pieces to be tested for, among other things, degradation of blood vessels caused by autoimmune disease. I was wrongly diagnosed with a rare autoimmune condition that's always deadly and has to be treated with such severe chemotherapy that it kills about forty percent of people who undergo it. Luckily, the mistake was caught in time, and I did not have this treatment.

The end result of these experiments was that no known treatment works on me. Unfortunately, this is not as unusual as you might think. The fact is that medical science doesn't understand headaches very well. And it also doesn't understand chronic pain very well, either. Progress is being made on both of these fronts, but we aren't there yet. In fact, it's entirely possible that "migraine" isn't a single kind of headache at all.

People often ask me what medications I'm taking. The answer is "none". I don't take anything for my migraines at all, and I also don't take any pain killers. Even if I have a bad head cold, I don't take medication that contains a pain killer. I've only taken pain killers once in the past five years -- that was when I got kicked by a horse. I took a little Advil for a day when that happened.

I'm not avoiding pain medication because I like being in pain. Quite the opposite. Chronic pain is very different from acute pain, and has to be treated differently. When you have acute pain -- for instance if you hit your head on something or have an occasional headache -- pain medications can be totally appropriate. But this isn't true for chronic pain. People like me who have chronic pain have a neurology that's overly sensitive to stimuli, and our central nervous system is too "ramped-up", so to speak. If we take anything that depresses our nervous system, we compensate for it over time. In particular, if we take pain medications for any significant length of time, we just become more sensitive to pain and we end up worse off than before.

So, counterintuitively, what we have to do is deliberately expose ourselves to stimulation that causes pain. The way we do this is by simply living our lives like anyone else. For example, I'm now sitting in a noisy coffee shop that's crowded, has rather harsh lighting, and also has a powerful smell of coffee. All of these are migraine triggers, and they exacerbate my chronic pain. But over the long run, environments like this will tend to cause my central nervous system to become less sensitive. This means less pain overall.

Besides living one's life, there are other coping strategies. One is to avoid thinking about it. This may sound trite, but simply getting in the habit of distracting oneself from the pain will actually make it better in the long run. General healthy living also helps a lot. This means exercising regularly, eating well, and keeping a regular sleep schedule. I'm better at some of these strategies than others, so there's always room for improvement.

As a result, I am in far less pain than I used to be, and I can live my life. People who meet me do not know that I have chronic migraines unless I tell them (which I usually don't, unless a particularly difficult day starts to impact me to a significant degree; then I might tell them what's happening).

It's important for someone in my situation to understand the causes of their chronic pain, despite the fact that the exact causes are not understood. We know there's a genetic factor, and we know some things that can exacerbate the condition. But in the end, we don't really understand why one person's pain becomes chronic while another person's doesn't. In my case, though, I'm very confident that my personality and lifestyle were major contributing factors. People with chronic pain tend to be "type-A" personalities; we're often very driven and obsessive. We tend to work too hard, keep irregular hours, become angry or frustrated too easily, and we're too demanding of ourselves. This describes me perfectly. For as long as I can remember, I've been totally obsessive about work (professional work as well as personal side-projects). I didn't sleep enough, I'd get stressed out to an irrational degree if my work wasn't going as well as I wanted, and I dealt with every difficulty by placing more demands on myself. I try really hard not to be like this, and I have varying degrees of success. I try to always ask myself, "how many people will die if this doesn't get done perfectly?". If the answer is "zero", I try to give myself permission to step back from the work. Because I am not planning on becoming a brain surgeon anytime soon, I expect the answer will always be "zero". But there will always be an obsessive freak living inside my brain, so this is always a challenge. In fact, one of the reasons I write this blog is to give that freak something to do that doesn't entail any obligations at all.

An unfortunate fact is that this type of personality tends to alienate people. But people with chronic pain need to avoid isolation. So people who develop chronic pain are often the very people who are most ill-equipped to deal with it. And if it's difficult to not be an obsessive person, it's (for me, at least) even harder not to become isolated. In my case, I'm sure that my wife quite literally saved my life. But the burden I placed on her was unbelievably heavy, not to mention unfair. People with chronic pain often do this. Our combination of personality and chronic pain will drive away the very people who want to help.

It would be wildly false if I were to claim that I've dealt perfectly with having chronic pain. But in a way, that's good news because I've been able to get on with my life despite my many flaws.

Thursday, March 19, 2015

What is Academic Philosophy, and Why is it So Sexist?

As a former academic philosopher, I saw a lot of sexist behavior on my profession. I was (perhaps naively) surprised by this. Of all the people in the world, I would have expected a group of highly-educated liberal academics to be more enlightened. I won't get into details here, but there's a growing consensus in academia that academic philosophy has a severe, pervasive, institutionalized problem with sexism. Women are underpaid, underrepresented in the profession, less likely to get tenure, less likely to get research support, and on and on. But why would a group of philosophers behave like this? Aren't philosophers supposed to be better than this?

As far as I'm aware, nobody has suggested that part of the reason why philosophy has a sexism problem is because of the nature of philosophy itself. Most people would probably say that any field would be just as likely to have this problem. But I think philosophy is special, in that the amorphous nature of the field provides very convenient cover for sexist behavior.

What is philosophy, anyway?
If you want to get a bunch of philosophers into an especially heated argument, ask them to describe what philosophy is. There are several standard answers to the question, "what is philosophy?". They include at least these:

  • The study of fundamental questions about the nature of reality, which are not addressed in the sciences.
  • The study of forms of reasoning and valid inference that are implicitly used in other fields, but which are not explicitly discussed within those fields.
  • An examination of the nature of knowledge, whether it's possible for humans to acquire knowledge, and if so, how.
  • The study of meaning and values, and how values can exist within a scientific worldview.

Subfields of philosophy are varied and numerous. Philosophers study questions that are as diverse as the interpretation of models from quantum mechanics; the nature of right and wrong; human action and intention; how economic models impact reasoning about social issues; what makes something a scientific law; texts of historical figures from thousands of years ago; formal logic and the foundations of mathematics; the nature of art and aesthetics; why the world exists rather than nothing; arguments for the existence of God; the nature of justice; whether laws describe reality or are merely conventions; and on and on. Any professional philosopher could probably rattle off at least a hundred specializations and subfields that are well-represented within academic philosophy.

Furthermore, all of these subfields and specializations have their own sub-branches. Philosophical topics are like a gigantic tree in which each question splits into several others repeatedly. For example, consider a field like epistemology, which is broadly defined as the study of knowledge. The number of branches of epistemology is truly staggering. But what's most interesting is that someone who specializes on one branch will likely believe that the people on other branches are fundamentally doing something misguided. For example, some philosophers believe that the study of knowledge is really a study of what we mean when we say that we "know" something. They are interested in understanding the language that we use to describe knowledge. Others believe that language is not really relevant here. They think that in order to understand knowledge, you should study the forms of reasoning that are most successful -- in other words, that you should study how science adds to our store of knowledge. Yet others believe that the very term "knowledge" is misleading. They think that epistemology is really about learning how to judge when to be confident of something, and understanding how you should update your beliefs when you acquire new evidence. Some believe that it's impossible to ever acquire knowledge in the first place, so the question is somewhat moot.

What's interesting here is that these various branches of epistemology are not just asking different questions -- they're actually in conflict with each other. If you believe that knowledge is impossible, then you're likely to think that your colleague who is studying scientific reasoning is really on the wrong track. If you think that understanding "knowledge" is really about language, then you'll probably think that your colleague who is busily constructing mathematical models of evidence-based reasoning is is also on the wrong track. In fact, one soon comes to the view that even within academic philosophy, and even within epistemology, we've batched together a set of projects that, at best, have nothing to do with each other. It's just a historical accident that we've given them the same general label, "epistemology".

Every field of philosophy is like this. Philosophy has a subfield called "philosophy of science", which studies philosophical questions arising from science. It is at least as diverse as the sciences themselves, and philosophers disagree vehemently with each other about what constitutes work that's relevant to the philosophy of science. It's common to hear someone derisively say, "He's not really doing philosophy of science; he's really more of a historian", just as it's easy to find someone who will say, "He's not a real epistemologist; he's a decision theorist."

And what do the major branches of philosophy have to do with each other, anyway? What does someone who is studying the interpretation of quantum mechanics have in common with someone who studies ancient Greek texts? Or what does someone who studies aesthetics have in common with someone who studies concepts of justice?

When I was an academic philosopher, I took the position that these subfields have nothing whatsoever of any significance in common with each other. I thought, and I still do, that it's just an accident that we happen to put these people in the same academic department. This really angered a lot of people. In fact, some of the most heated discussions (which weren't really discussions) I ever had were about the (boring) question of what should count as "philosophy". My former colleagues would invariably trot out some generalization that supposedly grouped together all these different areas of philosophy, but I always thought they did nothing but demonstrate my colleagues' appalling ignorance of what was happening outside their own cloistered department. Every single time they provided something that was supposed to differentiate philosophy from other fields, it was trivial to name another department (which was certainly not "philosophy"), in which that differentiating factor was on full display.

Academic philosophy is simply a loose agglomeration of different questions, methodologies, and topics. And there's nothing intrinsically wrong with that. If the questions are good ones, and the methodologies are sound, then by all means, pursue those problems. But the fact that your offices are in the same building doesn't make it the case that you're doing the same thing.

Academic philosophy and sexism
"Academic philosophy", because it has no unifying subject-matter, is like a blank canvas on which people paint their biases.

I'll illustrate this with a hypothetical example, but one which I've seen play out on numerous occasions, especially in my former department. The department has a vacancy for a new professor, and there is a huge number of highly-qualified applicants. It is more than reasonable for the department to think strategically about who they should interview, and who they should ultimately hire. Departments have their own strengths and weaknesses, and it is frequently the case that a department develops a perceived strength in one area of philosophy, such as ethics or the philosophy of science. A standard strategy for deciding who to interview or hire for a job is to try to find someone who will add to the department's strengths, while also mitigating the department's weaknesses. For example, if a department is perceived to be strong in the philosophy of science, a case can be made for hiring someone whose primary research interest is in that area, but who also has competence in another area for which the department is lacking. For example, if the department is weak in ethics, then it might make sense for the department to hire someone whose primary interest is in a subfield of philosophy of science, but who also conducts research or has a lot of teaching experience in ethics. Or the department might reverse those priorities, deciding to emphasize finding someone who is strong in ethics in order to shore up that weakness.

The word you always hear is "fit". Some compromise between the department's strengths and weaknesses is settled-upon, and an applicant who matches that compromise is a "fit" for the department.

The trouble is that, paradoxically, we're talking about whether someone "fits" into a structure that has no specific boundaries, and in which there is no agreement about those boundaries. This provides some very convenient cover for someone who just isn't very comfortable with treating a woman as his equal.

If you want to make an argument against hiring someone, the best way is often to show that they're not a good "fit". There are two ways to be a bad "fit" -- you can overlap too much, or not enough. So the trick is to find a plausible way to redraw your disciplinary boundaries so that the person falls too far inside or too far outside these imaginary boundaries. For example, suppose a department has two epistemologists, and there is a candidate who is also an epistemologist. Is the candidate a good "fit"? Let's say that the two epistemologists the department already has, as well as the job applicant, are similar in some ways but not in others (which will always be the case, of course). It's possible, with equal plausibility, to argue that the job candidate is too much like the current members of the department (thus being too close to count as a "fit"), too far away (also not a good "fit"), or even that the job candidate isn't even doing epistemology in the first place, because that line is arbitrary to begin with. A candidate who doesn't even consider him- or herself to be an epistemologist may be seen by a department as an epistemologist; or the candidate may be viewed as being "not a philosopher" -- the ultimate insult! In fact, it's quite easy for almost any academic philosopher to find another equally qualified academic philosopher who will claim that she is not actually working in the area that she thought she was.

This is emphatically not a hypothetical scenario. It happens all the time. When faculty are plotting and scheming about how to get their preferred person hired, the plots and schemes are typically about how to paint the undesirable candidate as not "really" a philosopher, or not "really" an ethicist, or not "really" a philosopher of science. I've heard plenty of people described as "really" a historian, or "really" a mathematician. Or, conversely, the scheme is to paint an undesirable candidate as too close to an current faculty member. The phrase one hears is, "Do we really need another faculty member who does the same thing as Professor so-and-so?". I've certainly been painted as "not really a philosopher" because my work often contained math, and I had gone so far as to actually collaborate with computer scientists. But as someone who thinks that "philosophy" is not a field of study at all, it never seemed improper to leave the department for an afternoon to have coffee with someone from the other side of campus.

But what does this have to do with sexism? The answer is that it provides fabulous cover for sex discrimination. When you're not comfortable with someone because of their gender, you can oppose their being hired, being awarded tenure, and so on by redrawing a set of disciplinary boundaries that were never real to begin with. In my experience, there are many victims of this strategy, but a disproportionate number of them are women.

The genius of this strategy is that when you redraw your discipline's boundaries in order to exclude someone, you are being no more arbitrary than when you redraw those boundaries to include someone else. The victim of this fraud is unable to point to any specific aspect of the hiring (or promotion) process that was unfairly or arbitrarily applied to her, because the entire process is a sham to begin with, from the moment a job advertisement is formulated that lists desired areas of expertise (those areas of expertise are, after all, themselves totally arbitrary). You can't point to a lie that was applied to your case when exactly similar lies were uniformly applied to everyone.

And of course, the perpetrators of this strategy have excellent cover. There are, after all, legitimate debates about what counts as "good" or "real" philosophy. So when a particular set of disciplinary boundaries are strategically chosen, there is an actual argument to be made that is capable of justifying that choice. Thus, someone who is engaged in sex discrimination can easily make an argument that he's only being rational, and that his judgment about "fit" is reasonable and principled.

Saturday, February 28, 2015

One (Very) Wealthy Person Could Reform Campaign Finance Today

It now costs over a billion dollars to make a credible run for President of the United States. How much more than a billion, we don't know, because so much secret money is given to groups that unofficially shill for a particular candidate. Which brings us to my favorite knock-knock joke, which I think came from Stephen Colbert:

"Knock-knock"
"Who's there?"
"Unlimited campaign contributions from corporations and unions"
"Unlimited campaign contributions from corporations and unions who?"
"I don't have to tell you who."

It's a big problem. When a candidate receives large donations from wealthy individuals, corporations, and unions, it's a form of legalized bribery. Everyone knows this. Furthermore, the vast amounts of money sloshing around campaign war chests makes it impossible for a third political party to emerge because any such party is caught in a chicken-and-the-egg problem. In order to attract big donations, the party would need to be credible. But in order to be credible, the party needs vast sums of money. So we have two bribed political parties who are incapable of passing meaningful campaign finance reform because they were put into office by the very people who benefit from the current, highly corrupt system. A game theorist would call this an "equilibrium". No change to the system will come about with an abrupt and severe shock to the entire system.

One might assume that an external shock large enough to disrupt the campaign finance/bribery system would have to be so gigantic that only the federal government could accomplish it. Billions upon billions of dollars has imparted a lot of inertia to the current system. Of course, a coordinated effort by millions of ordinary voters and taxpayers could accomplish it, too. But we now live in a society in which these ordinary people are so overwhelmed by economic demands on their time that no such grass-roots movement is likely to ever happen. In fact, I'm quite confident that it will never happen unless there's a massive, systemic economic or political collapse. This isn't out of the question, but it's not something we should hope for.

But there is a way out. And it would take only one (very) wealthy individual to make it happen. Such an individual would have to be a billionaire -- in fact, a multi-billionaire (although once you've made your first billion, the second billion is a lot easier). According to estimates, there are about five-hundred American billionaires. A small, well-coordinated group of extremely wealthy non-billionaires could do it, too. But for simplicity, I'll just talk about one (very) wealthy person.

In order to understand how to make this happen, we have to get clear on one simple fact, which is this: It doesn't matter, at all, how much money a political candidate has. What matters is how much money a political candidate has, relative to their opposition. This is because political campaign fund-raising is a classic "arms race" scenario. Just like in a literal arms race, advantages are always relative to the other side. If one side wages war by throwing rocks, and the other side has bows and arrows, then the latter has an unassailable advantage. If both sides have machine guns, then neither side has an advantage. The absolute strength of your weapons doesn't matter at all -- what matters is how your weapons compare to the other side's weapons.

This is where it would be possible for a (very) wealthy individual to disrupt the system. Individual donors, as well as corporations and unions, only give money to political candidates in order to give that candidate a relative advantage over the other side. This is why, when candidates appeal for donations, they emphasize how much money their opponent is raising; and they always portray themselves as falling behind in raising money. Individual donors give money in order to either level the playing field, or give their own candidate a relative advantage.

Now imagine that your favorite candidate is asking you for money. But imagine that you have an evil twin who is guaranteed to give an exactly equal amount to the opposing candidate. So if you're a Republican, your evil twin will give match your donations, dollar for dollar, by giving to the Democrats. Would you give money? Of course not! Your money would be wasted because it wouldn't provide a relative advantage to your preferred candidate.

Now imagine that you and your evil twin both have an evil uncle. Your evil uncle will match either of your contributions in the following way. At the end of each day, your evil uncle will estimate how much money you've given to the Republicans, and how much money your twin has given to the Democrats. If there's a difference, then your evil uncle donates the difference to the side that received the least amount of money. In this way, your evil uncle guarantees that any contributions you or your twin give are, in effect, impotent. Neither of your contributions end up making the slightest difference to your preferred candidates. You'd be an absolute fool to continue donating money. You'd keep your money in your pocket.

Let's think about what your uncle would have to do in order to pull this off. First, your uncle would have to be wealthier than you. If your uncle had only ten dollars to his name, you might donate twenty, overwhelm his bank account, and then continue giving more money.

Second, your uncle would have to be convincing. You'd have to really believe his threat. Most likely, you'd test him by giving some significant amount of money to your candidate and then you'd watch to see if he really gave the same amount to the opposing side. If he did carry out his threat, you'd probably seriously reconsider giving any additional money. You might "test" him again, maybe with more money. So your uncle would have to be resolute in his determination.

How much wealth does your evil uncle have to possess in order to prevent you and your twin from giving money to political candidates? At first glance, you might think that if you and your twin were each to give a few thousand dollars to each candidate, then he'd need enough money to match both sets of contributions. But that's not the case at all. In the worst case scenario, he'd need the difference between the two -- remember that your uncle is only trying to equalize the total contributions. A much smaller amount of money is necessary in order to do that.

But that's the worst case scenario. If your uncle were to do this from the very beginning of the campaign season, he'd probably change your behavior pretty quickly. After you and your twin had "tested" your uncle and determined that he'll carry out his threat, you'll stop wasting your money. Unless you're both stupid, you and your twin will stop giving money to these campaigns very quickly. In the end, your uncle will have to have a lot of money in his bank account to make his threat credible, but he won't end up spending very much of it.

Now suppose that you and your twin are irritated by your uncle's meddling behavior and you want to put an end to it. How do you do that? The short answer is that you can't. Here's why.

The only way to end his meddling is by making it prohibitively expensive for him. The only way to do that is to create a large imbalance in the amounts that you and your twin are donating. So you have to agree that one of your candidates will be given a lot of money until your uncle's bank account runs out. Now, let's assume that somehow, you've actually scraped together enough money to make this possible. But now you face the question of selecting which candidate will get this money, and you can't possibly agree to that, because neither of you will permit the opposing candidate to have an advantage. For instance, if you're the Republican, would you agree that the Democrat should be given so much money that your uncle goes bankrupt? Of course not! And the same holds true if you're the Democrat.

Probably, what you'd actually do, if you were fanatical enough, would be to launder your money into the coffers of your candidate so that your uncle wouldn't find out about it. Let's assume that you could do this legally. It wouldn't matter, because the candidate's campaign has to spend the money in order for the donation to make a difference, and your uncle could easily estimate how much money is being spent. Of course, your uncle would never be able -- even under the best circumstances -- to be 100% accurate in his estimates, but that doesn't matter. He'd probably overestimate sometimes to the benefit of one candidate, and underestimate other times to the benefit of the other. The net effect would be close enough to zero that it wouldn't make a difference.

The fact of the matter is that your evil uncle has you checkmated. The only way to undermine him is to coordinate your efforts; but you can't do that because you're on opposite sides. You'd spend your money on something else. Maybe even something worthwhile.

All we need is for one (very) wealthy person (or a small group of somewhat less wealthy individuals) to be our collective evil uncle. The final element of the plan is that our uncle would also pledge not to match any money obtained by public financing. After a relatively short, but tumultuous time, there would only be public financing left. Or perhaps our uncle could pledge not to match small donations of less than some pre-specified amount. Either way, our American system of legalized bribery would end. Both parties would end up strengthened because they could spend their efforts actually doing strange things such as communicating with voters; elected officials wouldn't have to spend their time on the phone begging for money; and there would be a real chance of a third political party emerging.