About Us
Opportunities
Research Fellow
By Research Fellow Eliezer Yudkowsky, your potential coworker
Suppose a Bayesian decision agent, a classical expected utility maximizer, had the ability to modify her own source code – including the part of herself that chooses how to modify source code. When you plug this dilemma into classical Bayesian decision theory, it barfs on an infinite recursion. You can use classical decision theory to choose between actions, and choose between source code that chooses between actions, but you can't actually close the loop; classical decision systems can't quine themselves.
This is one of the many fundamental open problems required to build a recursively self-improving Artificial Intelligence with a stable motivational system. Now, if you're the person we're looking for, you can probably look at the above problem and think of a clever ad-hoc solution off the top of your head. So you need to be adaptable, and a fast unlearner, because cleverness is one of many habits of thought you'll need to unlearn. We're not looking for an ad-hoc solution. This isn't about pumping out another paper or finding a quick hack that gets the job done. Too much weight is going to rest on this. Anything we don't understand has to be solved, not cleverly swept under a rug. I'm not looking for someone who can invent powerful tools, like neural networks or evolutionary programming. I'm looking for someone who can help create new basic foundations. Pretend you're working in a historical epoch before anyone realized that math could describe the business of "gathering evidence" or "betting on games of chance", and ask yourself how you'd go about inventing Bayesian probability theory or Bayesian decision theory. The task is to illuminate the underlying structure of cognitive processes that are currently murky and ill-defined. Note that this is a matter of applied math, not math that is beautiful solely for the sake of being beautiful – the math has to describe an AI.
So what does it take to get that job done? Well, for starters, sheer raw fluid intelligence, plain old-fashioned Spearman's g. You'll need to know things that aren't in textbooks and apply skills that aren't taught in classes. You'll have to pick things up rapidly, from a few hints, without them being hammered into you. I attended the inaugural symposium of the Redwood Center for Theoretical Neuroscience, and they asked a panel of prestigious experimental neuroscientists what kind of experience they'd most like to see in a hiree. And one said "Neuroscience", and one said "Electrical engineering", and then one said, "I'd rather hire a physicist, because they can learn anything," and the rest all nodded. That's the indispensable quality we're looking for, whether it appears in a physicist or not.
It would be extremely helpful if you've already studied Bayesian probability theory, Bayesian decision theory, and several different variants of mathematical logic. But if you can learn those fields in a couple of weeks, that's impressive too. You do need to demonstrate pre-existing mathematical competence at something or other – mainly because I don't want to deal with applications from people who think they've discovered the one great key principle that underlies all intelligence, but who never did get the hang of algebra. You also need to be able to think in code; eventually all of this has to translate into a computer-bourne physical process – that's the anchor against which all questions and answers take place. You need an instinct for what code can be made to do.
You'll be the Singularity Institute's second Research Fellow, and I'm on the lookout for someone whose abilities complement my own. I once had an exchange which sticks in my mind, and illustrates this point fairly well. I was pondering utility functions, and said: "Utility functions are unique up to a positive affine transformation; what kind of information does that preserve? It preserves ordering, but it's more than just that. It doesn't preserve proportions..." And the one who was listening, acting as my person-to-bounce-ideas-off-of, said, "It preserves relative intervals." And lo, I immediately knew exactly what it meant that the information in a utility function consisted of proportions between intervals between outcomes. (Left as an exercise to the reader.) The most difficult and important part of this conversation was knowing which question to ask at the start of it. The second most important part was knowing what the answer meant. Nonetheless, it was still the other person who supplied the answer, because that person had studied a broader range of math than I had. In the process of learning which questions to ask and what the answers meant, I had to study a whole big bunch o' stuff. Evolutionary biology doesn't have any obvious relevance to building a reflective AI, making a decision system that quines itself - until you realize that evolutionary biology describes the mathematics of an optimization process which is much simpler than human intelligence, and therefore more studiable; and moreover, natural selection is an optimization process that pre-1960s biologists often anthropomorphized, and therefore evolutionary biologists have a whole minor industry devoted to stamping out anthropomorphic thinking about their nonhuman, mathematically describable optimization process. But the flip side of this is that any time I spent studying things like evolutionary biology, evolutionary psychology, neuroscience, cognitive psychology, heuristics and biases, etcetera etcetera, I did not spend studying math, and so I did not know off the top of my head that an affine transformation preserves relative intervals.
By dint of studying this diversity of disciplines and by banging my head against the problem for years, I was eventually able to get my bearings (in 2003; don't trust anything I wrote before then). This self-teaching process was painful, and as SIAI's second Research Fellow, you are not expected to go through it. Instead you will be expected to draw on my own sense of which questions need to be asked, what the answers mean, which avenues of investigation are blind alleys and which are likely to produce productive insights, which solutions are ad-hoc and which may lead into something generalizable. Since you don't require all those other fields, I would like SIAI's second Research Fellow to have more mathematical breadth and depth than myself. Breadth is more important than depth; I don't currently foresee requiring the Deep Math of any single existing subfield, even existing decision theory.
The most painful part of working with me, from an emotional standpoint, is likely to be when I reject a solution you offer. Expect that to happen a lot at first. The space of solutions that lead somewhere is a tiny target in the space of solutions that initially look attractive. Trying to come to terms with reflectivity means chasing the tail of the lizard's tail, always asking how you yourself produced the clever solution you just came up with. Another painful aspect is that we may not solve most of the problems we tackle - I count it a success if we can get one good, reusable insight before we come to a blank wall and have to move on. And there is finally the patience to stare silently at a blank sheet of paper, thinking in circles, for hours, until drops of blood form on your forehead, most of the time. This is a job that involves solid hours of thinking without clear information on what to do next.
Eventually, if you have high enough fluid intelligence, you'll pick up the pattern and wield the art for yourself, rather than following my movements; but I do expect there to be a training period while you follow my movements. Perhaps that counts as "communication skills" or "people skills" or something. But it is a strictly rationalist social skill; a strictly nerdish form of emotional maturity, whose objective is solely to solve a technical problem. You will be expected to communicate only with a fellow genius working on the same challenge. I can be the speaker-to-nontechies of our pair. Any people skills you have beyond this are a bonus, but they are not required.
We don't dare hire a dysfunctional genius – it's not safe – but if you're so far outside the mainstream that you can't see the mainstream on a clear day with a radio telescope, that's fine so long as you get the job done. I don't give a damn if you can only do your best thinking while wearing a clown suit. If you can only do your best thinking while yodeling, that's a problem, because I can't get much thinking done sitting next to someone who's yodeling.
Don't expect fame or fortune. The Singularity Institute is not your employer, and we are not paying you to accomplish our work. The so-called "Singularity Institute" is a group of humans who got together to accomplish work they deemed important to the human species, and some of them went off to do fundraising so the other ones could get paid enough to live on. Don't even dream of being paid what you're worth, if you're worth enough to solve this class of problem. As for fame, we are trying to do something that is daring far beyond the level of daring that is just exactly daring enough to be seen academically as sexy and transgressive and courageous, so working here may even count against you on your resume. But that's not important, because this is a lifetime commitment. Let me repeat that again: Once you're in, really in, you stay. I can't afford to start over training a new Research Fellow. We can't afford to have you leave in the middle of The Project. It's Singularity or bust. If you look like a good candidate, we'll probably bring you in for a trial month, or something like that, to see if we can work well together. But please do consider that, once you've been in for long enough, I'll be damned hurt – and far more importantly, The Project will be hurt – if you leave. This is a very difficult thing that we of the Singularity Institute are attempting – some of us have been working on it since long before there was enough money to pay us, and some of us still aren't getting paid. The motivation to do this thing, to accomplish this impossible feat, has to come from within you; and be glad that someone is paying you enough to live on while you do it. It can't be the job that you took to make the rent. That's not how the research branch of the Singularity Institute works. It's not who we are.
Positive aspects of the job: You can spend as much time as you feel you need learning new things and levelling up your math and science skills; you don't have to explain anything to your nontechie boss; you don't have to teach or pump out papers; no one cares if you wear a clown suit; you get to save the world.
Don't think that this whole long description has to fit you exactly. I am merely trying to convey some idea of what the job entails. Are you a math talent who can think in code? Are you incredibly brilliant? Do you want to save the world? Drop me an email at yudkowsky@singinst.org and put JOB: in the subject line.
Please remember that, at this present time, we are looking for breadth of mathematical experience, not coding skill as such.