What you should not do, I think, is worry about the opinion of anyone beyond your friends. You shouldn’t worry about prestige. Prestige is the opinion of the rest of the world.

[…]

Prestige is like a powerful magnet that warps even your beliefs about what you enjoy. It causes you to work not on what you like, but what you’d like to like.

[…]

Prestige is just fossilized inspiration. If you do anything well enough, you’ll make it prestigious. Plenty of things we now consider prestigious were anything but at first. Jazz comes to mind—though almost any established art form would do. So just do what you like, and let prestige take care of itself.

Prestige is especially dangerous to the ambitious. If you want to make ambitious people waste their time on errands, the way to do it is to bait the hook with prestige. That’s the recipe for getting people to give talks, write forewords, serve on committees, be department heads, and so on. It might be a good rule simply to avoid any prestigious task. If it didn’t suck, they wouldn’t have had to make it prestigious.

A great article by Paul Graham here.