Using AI for developer education | RareCode by RareSkills

AI has had such a huge impact in the development of software in such a short time that many developers fear it will take their jobs.

After all, if a tool can produce in minutes what used to take days, it would seem that the human developer is going the way of the horse and carriage.

In practice however, AI is only able to speed up mundane work that has been done so many times that the AI has seen thousands of examples of such a problem on the internet. Once the AI has to deal with novel technology stacks, it quickly breaks down.

Whether AI is eventually able to overcome this limitation remains to be seen.

However, one major unexplored area for AI is developer education.

Ask any junior developer what they want most in a job, and most will respond saying "I want good mentorship from a senior developer" -- some will even value this over an otherwise higher salary, all else being equal.

With the right framework, AI can serve the role of a senior engineer mentoring a junior one.

In a typical mentorship relationship, the senior engineer assigns a junior engineer a tasks, the junior engineer struggles with the task and asks the senior engineer for help when they get stuck.

After they complete the task, the senior engineer gives feedback about how to make a good solution even better.

Here's what makes this learning environment so effective:

• the junior gets feedback while the problem is still fresh in their mind
• the junior doesn't waste time watching a lecture where they may or may not remember what the professor taught a couple hours afterwards
• the knowledge gaps of the junior get efficiently exposed, so the junior can focus on areas they are weak in

Due to the high risk of training junior engineers (who may leave for higher pay), this arrangement is not as common as it used to be when even junior engineers were in demand.

Nonetheless, this model shows what peak education looks like as opposed to the alternatives:

In a traditional school setting, students are lectured about some material, then they practice it several hours later when they've already forgotten it. This doubles the time they spend to achieve the same outcome.

In an online course, students watch videos yet rarely retain the information because there is little accountability or structure -- if any -- to practice what was taught.

At RareSkills, we tried to replicate the junior-senior mentorship model as closely as possible. The "lectures" are usually not just the instructor spewing out information, but actively challenging the student's understanding of the subject.

Then, the instructors make themselves available for answering questions that come up and for reviewing the completed assignments one-on-one.

One insight we gained was that around late 2024, the volume of questions that our instructors received dropped suddenly -- this coincided with the release of very powerful AI models that could give very nuanced answers to specific questions instead of pattern-matching a question to an answer.

AI has been around for some time of course, but only recently has it gotten to the point of being better than humans at some tasks.

The next natural question of "how far can we push the AI-as-a-mentor" model?

Quite far it turns out.

Nowadays, one could learn a new programming language by opening a chat window and asking the AI to teach the subject.

For this to be a success however,

1. The student needs to ask the right questions
2. The student needs to practice what they learn

AI in its current state isn't able to do those things. Instead,

1. We create an environment where students are presented with problems that prompts the student to ask the right questions.
2. We pre-create the long list of problems the student will practice (this eliminates the possibility that the AI would hallucinate creating a practice problem that drills something the student hasn't learned yet).

As long as each practice problem is "close enough" to "within reach" of the student's current capabilities, then the student knows which questions they should be asking the AI.

We created a Rust Programming Course based on this these insights and hosted it on RareCode.ai.

Some students who used RareCode wrote hundreds of lines of Rust in less than a week -- that's more than some engineers push to production in the same time -- but consider that the student is using Rust for the first time!

The nice thing about AI-as-an-instructor is that the lessons can be extremely short and the students can spend close to 99% of their time directly practicing the skill instead of reading about it. If the lesson missed a key point, the student would naturally ask the AI about it when they encounter errors in their code.

Of course, we understand the limits of this approach. AI only gives good answers on topics it has seen a lot of examples of. We still run our human-led cohorts.

However, for subjects the AI knows well, the loop of:

brief instruction -> solve problem -> AI feedback or ask AI if stuck

probably cannot be improved upon with any other existing solution.

A human instructor cannot have the stamina to keep giving feedback and filling in knowledge-gaps to the same student for several hours in one day. Even if the instructor could such an arrangement would be prohibitively expensive.

Blockchain is a fast-moving field and we need human instructors to teach things the AI hasn't learned yet.

However, a significant amount of what a student needs to learn is well-represented in what AIs already know well.

The student and the AI just need a good framework to get the information out of the AIs weights and into the student's brain. That's what RareCode provides.