One day a number of years ago, I was deep into a game of draughts on holiday with my daughter, then almost four, in the small library of a beachfront town. Her eye drifted to a nearby table, where a black-and-white board bristled with far more interesting figures (many a future chess master has been innocently drawn in by “horses” and “castles”).

“What’s that?” she asked. “Chess,” I replied. “Can we play?” she pleaded. I nodded absently.

There was just one problem: I didn’t know how. I dimly remembered having learned the basic moves as a kid, but chess had never stuck. This fact vaguely haunted me through my life. I would see an idle board in a hotel lobby or a puzzle in a weekend newspaper supplement, and feel a pang.

I had picked up a general awareness of chess. I knew the names Bobby Fischer and Garry Kasparov. I knew that the game had enchanted historical luminaries including Marcel Duchamp and Vladimir Nabokov. I knew the cliche about grandmasters being able to look a dozen moves ahead. I knew that chess, like classical music, was shorthand in movies for genius – often of the evil variety. But I knew chess the way I “knew” the Japanese language: what it looks like, what it sounds like, its Japaneseness, without actually comprehending it.

I decided to learn the game, if only to be able to teach my daughter.

It took a few hours, hunched over my smartphone at kids’ birthday parties or waiting in line at Trader Joe’s, to get a feel for the basic moves. Soon, I was playing, and some-times even beating, the weakest computer opponents (the ones with catastrophic blunders abundantly programmed in). Yet it soon became apparent that I had little concept of the larger strategies. I didn’t want to try to teach what I knew only poorly.

And yet, how to learn? The number of chess books was dauntingly huge. Sure, there was Chess for Dummies. But beyond that, the chess literature was enormous. It was filled with algebraic-looking thickets of chess notation, a quasi-language that itself had to be learned. And the books were achingly specific: for example, A Complete Guide to Playing 3 Nc3 Against the French Defence.

That’s right: an entire book devoted to the permutations of a single move – a move that, I should add, has been regularly played for a century. Yet people were still figuring out, 100 years and many chess books later – 288 pages’ worth of new things to say about it.

A well travelled fact that one hears early in chess is that after only three moves, there are more possible game variations than there are atoms in the universe. And, indeed, I felt cosmically stupefied as I tried to figure out how to boil down this exponentially complex game to someone whose favourite show was Curious George.

So I did what any self-respecting modern parent does: I hired a coach. The twist was that I wanted someone to teach my daughter and me at the same time.

For most of us, the beginner stage is something to be got through as quickly as possible, like a socially awkward skin condition. But even if we’re only passing through, we should pay particular attention to this moment. For once it goes, it’s hard to get back.

Think of a time when you first visited a new, distant place, one with which you were barely familiar. Upon arrival, you were alive to every novelty. The smell of the food in the street! The curious traffic signs! The sound of the call to prayer! Flushed from the comfort of your usual surrounds, forced to learn new rituals and ways to communicate, you gained sensory superpowers. You paid attention to everything because you didn’t even know what you needed to know to get by. After a few days, as you became more expert in the place, what seemed strange began to become familiar. You began noticing less. You became safer in your knowledge. Your behaviour became more automatic.

Even as your skills and knowledge progress, there is a potential value to holding on to that beginner’s mind. In what’s come to be known as the Dunning-Kruger effect, the psychologists David Dunning and Justin Kruger showed that on various cognitive tests the people who did the worst were also the ones who most “grossly overestimated” their actual performance. They were “unskilled and unaware of it”.

This can certainly be a stumbling block for beginners. But additional research later showed that the only thing worse than hardly knowing anything was knowing a little bit more. This pattern appears in the real world: doctors learning a spinal surgery technique committed the most errors not on the first or second try, but on the 15th; pilot errors, meanwhile, seem to peak not in the earliest stages but after about 800 hours of flight time.

I’m not suggesting experts have much to fear from beginners. Experts, who tend to be “skilled, and aware of it”, are much more efficient in their problem-solving processes, more efficient in their movement (the best chess players, for example, tend also to be the best speed-chess players). They can draw upon more experience, and more finely honed reflexes. Beginner chess players will waste time considering a huge range of possible moves, while grandmasters zero in on the most relevant options (even if they then spend a lot of time calculating which of those moves are best).

And yet, sometimes, the “habits of the expert”, as the Zen master Suzuki called it, can be an obstacle – particularly when new solutions are demanded. With all their experience, experts can come to see what they expect to see. Chess experts can become so entranced by a move they remember from a previous game that they miss a more optimal move on a different part of the board.

This tendency for people to default to the familiar, even in the face of a more optimal novel solution, has been termed the Einstellung effect (after a German word that means “set”).

In the famous “candle problem”, people are asked to attach a candle to the wall using nothing more than a box of matches and a box of tacks. People struggle to solve it because they get hung up on the “functional fixedness” of the box as a container for tacks, not as a theoretical shelf for the candle. There is one group, it turns out, that tends to do pretty well on the candle problem: five-year-olds.

Why? The researchers who found this suggest that younger children have a more fluid “conception of function” than older children or adults. They are less hung up on things being for something, and more able to view them simply as things to be used in all sorts of ways. Small wonder they conquer new technology so handily; everything is new for them.

Children, in a very real sense, have beginners’ minds, open to wider possibilities. They see the world with fresher eyes, are less burdened with preconception and past experience, and are less guided by what they know to be true.

They are more likely to pick up details that adults might discard as irrelevant. Because they’re less concerned with being wrong or looking foolish, children often ask questions that adults won’t ask.

No one wants to stay a beginner. We all want to get better. But even as our skills improve, and our knowledge and experience grow, what I hope to encourage is the preservation, or even cultivation, of that spirit of the novice: the naive optimism, the hypervigilant alertness that comes with novelty and insecurity, the willingness to look foolish, and the permission to ask obvious questions – the unencumbered beginner’s mind.

What the chess master Benjamin Blumenfeld advised a century ago applies as much to life as to chess: “Before you make your move, look at the position as if you were a beginner.”

When my daughter first began competing in school chess tournaments, I would often chat to other parents. Sometimes, I’d ask if they played chess themselves. Usually, the reply was an apologetic shrug and a smile. When I volunteered that I was learning to play, the tone was cheerily patronising: “Good luck with that!” I thought: “If this game is so good for kids, why are adults ignoring it?” Seeing someone playing Angry Birds, I wanted to tap them on the shoulder and say: “Why are you having your kids do chess while you do that? This is the game of kings! There are chess games recorded from the 15th century!”

At chess tournaments, I saw a dynamic that was all too familiar from the world of children’s activities: kids doing the activity, and adults like me staring into their smartphones.

Sure, we parents had work to do, work that we allowed to spill into weekends, work that helped pay for the lessons our kids were enjoying (or enduring). But I also wondered if we, in our constant chaperoning of these lessons, were imparting a subtle lesson: that learning was for the young.

Strolling down the hall during one tournament, I looked into a classroom and saw a group of parents with what I took to be an instructor. They were playing chess! Just then, as if on cue, a group of kids passed me, peering in on the same scene. “Why are adults learning chess?” one asked, in a vaguely mocking tone, to the collective amusement of the group. They marched on while I slowly died in front of a cheery bulletin board.

I was tired of sitting on the sidelines. I wanted in. And that is how I got a membership card from the US Chess Federation and started throwing myself in.

Early on, I was nervous, even though I really had nothing to lose, save my pride. “A master can sometimes play badly,” as one grandmaster put it, “a fan, never!” And fan I was: the sombre rituals, the pulse-pounding encounters, the tense atmosphere. It was three hours of sustained concentration and intense thinking, with my phone turned off. It felt like a gym for the brain.

Being a beginner can be hard at any age, but it gets harder as you get older. Children’s brains and bodies are built for doing, failing, and doing again. We applaud virtually anything they do, because they are trying.

With adults, it’s more complicated. The phrase “adult beginner” has an air of gentle pity. It reeks of obligatory retraining seminars and uncomfortable chairs. It implies the learning of something that you should have perhaps already learned.

There is safety in sticking with what we’re already good at. “It’s hard to be old and bad at something,” as a friend, returning to hockey after many decades, put it. We can be so put off by being a beginner that we forget we were once beginners in all sorts of things, until we were not.

Adult beginners face their own version of what is known by sporting coaches as “stereotype threat”, whereby a negative image becomes associated with a particular group of players, and leads them to repeat mistakes – in this case, one that says it’s harder to learn when you’re older. There’s a pernicious, goading little voice: “You’ve started too late. Why bother?” One day, at her swimming lesson, I was impressed to see my daughter “flip turn” at the end of the lane while doing a backstroke. This is not something I can do. “How’d you learn to do that?” I asked. “You have to be a kid,” she responded matter-of-factly.

As I was finding out, this kind of idea is deeply ingrained in chess. There seems to be a relation between the age at which you first learned the game and your later success in tournaments. This idea is so pervasive that Magnus Carlsen, the current No 1, is held as a fascinating outlier. “At five years old,” one account marvels, “an age by which any aspiring grandmaster should at least have made a start, Magnus Carlsen showed little interest in chess.”

Sitting down against younger opponents, I tried to keep in mind a bit of advice gleaned from the Guardian writer Stephen Moss’s book The Rookie: just face them the way you would anyone else.

This could be hard. The way they played just threw me. In the face of my agonised dithering, they would launch fast, brute-force attacks – sometimes effective, sometimes foolhardy. “Children just kind of go for it,” Daniel King, the English grandmaster and chess commentator, told me. “That kind of confidence can be very disconcerting for the opponent.”

Young children, for example, have been shown to be faster and more accurate at tests involving “probabilistic sequence learning” – the sort in which people must guess which triggers will lead to what events (for example, if you press button A, event X will happen).

After the age of 12, this ability begins to decline. As researchers suggest, people start relying more on “internal models” of cognition and reasoning, instead of what they see right in front of them. In other words, they overthink things. In chess games, where my adult opponents often seemed to battle unseen internal demons, the kids just seemed to twitch out a series of moves.

I was buying into the stereotype threat. If I lost to an adult, I would chalk it up to my own stupid errors. But if I lost to a child, I would suddenly imagine them as some incipient genius against whom I never had a chance.

When I asked our chess coach about what it was like to teach adult chess beginners as opposed to child chess beginners, he thought for a moment and said: “Adults need to explain to themselves why they play what they play.” Kids, he said, “don’t do that”. He compared it to languages: “Beginner adults learn the rules of grammar and pronunciation and use those to put sentences together. Little kids learn languages by talking.”

The analogy goes deeper than we might think. My daughter was, in effect, learning chess like a first language, whereas I was learning it like a second language. Even more important, she was learning it young.

Language is one of those endeavours (like music, and perhaps chess) that seems to flourish best if learned during a so-called “sensitive period” in which, as one researcher has described it, “neural systems are particularly responsive to relevant stimuli, and are more susceptible to change when stimulated”.

By contrast, because I am an adult, expert speaker of English, my brain may be so “tuned” to the sounds of my native language that it is harder for me to take on new grammar. What I know already gets in the way of what I want to learn. Kids, by knowing less, can actually learn more (the cognitive scientist Elissa Newport calls it the “less-is-more hypothesis”).

Harder does not mean impossible. “Sensitive” periods are not “critical” periods, and the science, in any case, is not conclusive. The skill of having perfect pitch, for example, which is not only exceedingly rare but has long been thought to be impossible outside a narrow age range in childhood, can be trained in some adults, as research from the University of Chicago has shown (albeit to not quite as high a level as those possessing “true” perfect pitch).

Kids often make more progress simply because they are kids, with lives built largely around learning, having few other responsibilities, and with eager parents to cheer them on. They are also motivated: if you were dropped into an entirely new setting, the way infants are, and found that you couldn’t communicate, you’d probably learn pretty quickly.

You may, by now, be rightly asking, why should I bother learning a bunch of things that aren’t relevant to my career? Why dabble in mere hobbies when I’m scrambling to keep up with the demands of a rapidly changing workplace?

First, I might suggest that it’s not at all clear that learning something like singing or drawing actually won’t help you in your job – even if it’s not immediately obvious how.

Learning has been proposed as an effective response to stress in one’s job. By enlarging one’s sense of self, and perhaps equipping us with new capabilities, learning becomes a “stress buffer”.

Claude Shannon, the brilliant MIT polymath who helped invent the digital world in which we live today, plunged into all kinds of pursuits, from juggling to poetry to designing the first wearable computer. “Time and time again,” noted his biographer, “he pursued projects that might have caused others embarrassment, engaged questions that seemed trivial or minor, then managed to wring breakthroughs out of them.”

Regularly stepping out of our comfort zones, at this historical moment, just feels like life practice. The fast pace of technological change turns us all, in a sense, into “perpetual novices”, always on the upward slope of learning, our knowledge constantly requiring upgrades, like our phones. Few of us can channel our undivided attention into a lifelong craft. Even if we keep the same job, the required skills change. The more willing we are to be brave beginners, the better. As Ravi Kumar, president of the IT giant Infosys, described it: “You have to learn to learn, learn to unlearn, and learn to re-learn.”

Second, it’s just good for you. I don’t mean only the things themselves – the singing or the drawing or the surfing – are good for you (although they are, in ways I’ll return to). I mean that skill learning itself is good for you.

It scarcely matters what it is – tying nautical knots or throwing pottery. Learning something new and challenging, particularly with a group, has proven benefits for the “novelty-seeking machine” that is the brain. Because novelty itself seems to trigger learning, learning various new things at once might be even better. A study that had adults aged 58 to 86 simultaneously take multiple classes – ranging from Spanish to music composition to painting – found that after just a few months, the learners had improved not only at Spanish or painting, but on a battery of cognitive tests. They’d rolled back the odometers in their brains by some 30 years, doing better on the tests than a control group who took no classes.

They’d changed in other ways, too: they felt more confident, they were pleasantly surprised by their work, and they kept getting together after the study ended.

Skill learning seems to be additive; it’s not only about the skill. A study that looked at young children who had taken swimming lessons found benefits beyond swimming. The swimmers were better at a number of other physical tests, such as grasping or hand-eye coordination, than non-swimmers. They also did better on reading and mathematical reasoning tests than non-swimmers, even accounting for factors such as socio-economic status.

Many of these studies or recommendations are oriented toward children. Chess, for example, is held up as a way to improve children’s focus and concentration, to strengthen their problem-solving skills, to bolster their creative thinking. But I’ve become convinced that whenever something is touted as being good for children, it’s even better for adults, in part because we assume we no longer need all those benefits an activity is said to provide.

And yet what better remedy for the widespread affliction of “smartphone addiction” than two hours of burning your eyes and brain into 64 squares on a board, trying to analyse an almost infinite variety of moves and countermoves?

Learning new skills also changes the way you think, or the way you see the world. Learning to sing changes the way you listen to music, while learning to draw is a striking tutorial on the human visual system. Learning to weld is a crash course in physics and metallurgy. You learn to surf and suddenly you find yourself interested in tide tables and storm systems and the hydrodynamics of waves. Your world got bigger because you did.

Last, if humans seem to crave novelty, and novelty helps us learn, one thing that learning does is equip us with how to better handle future novelty. “More than any other animal, we human beings depend on our ability to learn,” the psychologist Alison Gopnik has observed. “Our large brain and powerful learning abilities evolved, most of all, to deal with change.” We’re always flipping between small moments of incompetence and mastery. Sometimes, we cautiously try to work out how we’re going to do something new.

Sometimes, we read a book or look for an instructional video. Sometimes, we just have to plunge in.

• This is an edited extract from Beginners: The Curious Power of Lifelong Learning by Tom Vanderbilt is published by Atlantic and available at guardianbookshop.co.uk