libfpv - FPV Drone in Any Game

Hi there, it's been a long time since I worked on a really interesting project. Happy to have you here, it seems like you're really interested in how libfpv was made.

Spoiler, you might be deceived.

Contents

Demonstration

First of all, I have to show off what libfpv is capable of. You, yes you, come on, play the Mario 64 right here, load a game (space, space), wait until Mario appears and press F to spawn your drone:

If you need the fullscreen experience go to this URL: https://sm64ex-fpv.hammerill.com

I also did already present the project on Reddit, see the very first post with Super Mario 64:

And the next one with Grand Theft Auto IV:

Finally, here goes my full playthrough of SM64 using almost exclusively the quad:

Context

All started when I began to be interested in FPV simulators in general. First the dedicated ones like Liftoff and others, then the FPV mods for large games like GTA V. I won't tell why I was interested in all this in the first place, it was kinda emotional... I'll just skip this.

Anyways, playing GTA V with the FPV mod was excellent, it was like if I were discovering the game from completely another angle. Also the feel of freedom: normal FPV sims force you to stay within map limits but with this mod I could fly all around the whole GTA V map, with all its buildings, nature and whatever else.

That was a nice experience, and it made me buy an actual FPV drone in real life. Today you can buy the smallest tinywhoops possible, so I got myself an Air65 II. I want to solder my own 5" later but I ain't there yet (also I suck at soldering things).

Flying this little dude made me understand one thing: I totally skipped learning the "drone arming" part while playing the FPV mod. I just disabled it and never thought about it. Yet that's the thing you do all the time with an actual drone, and it made me spend minutes with all the stuff bound before taking off because I just didn't know I had to arm the quad and how was I supposed to do so... Writing this down feels stupid but I really was in this situation.

This was the first thing that made me concerned about the fact that the mod I was playing does omit things that happen IRL, so it surely doesn't simulate something else.

By that time I also started customizing the OSD of my drone, put my battery level there, signal level there, pilot name HAMMERILL down there... And then be able to hide it all with an OSD disable switch on my radio because I don't really have to see it 100% of the time.

Also, about the switches. In the very beginning, each time my whoop fell to the ground upside down I had to take off my goggles and come flip it manually. Then I saw some guy online doing something magical: a CRASH FLIP and then the drone flips by itself. I then learned that it's an actual feature called turtle mode. Again, that's something the simulator didn't teach me.

Activating turtle mode is very technical: while in the sim you just click "flip" and voilà your drone flies again, IRL you have to disarm, put the crash flip switch, arm with it (ensuring throttle 0%), push the stick in the direction to flip, disarm, put off the same switch, arm (ensuring throttle 0% again) and take off. As of today, I never played an FPV sim which makes you go through this whole procedure so you could get used to it even before piloting an actual drone.

...

After all that, I also wanted to use an FPV drone to record things in BeamNG.drive, especially being able to do so on a scene recording so I could record myself drifting post factum. So I've downloaded the FPV drone mod and... I couldn't find how to spawn it. I only later understood that the quad is a new "vehicle" so I spawned it that way. I've bound my radio but couldn't find any option to adjust my rates. Finally I took off and... The physics were horrible, it isn't the same way the other simulators worked and not at all how my whoop flies. This was horrible, the drone was unresponsive, didn't retain its direction (normally PID in real life is made to make the drone feel responsive) and anything.

I didn't even try to spawn it inside a scene recording, I wouldn't just know how to do so I could spawn the FPV "vehicle" inside it and fly around the scene (undocumented mod), and I wasn't interested to do so with these physics.

The Two Issues of FPV Mods

All this made me understand that the "FPV mods" in general have two issues that matter to me. Here they go:

TLDR: the modders always have to make the whole FPV mod from scratch and it won't be as realistic as it could be.

Idea

One day I just had a funny thought: "I would like to play an FPV sim where the quad just... runs on Betaflight, so it's like an actual drone flying literally across the virtual world, and so I can tune it with Configurator and all that".

Little did I know, after searching that I would actually fall upon pr0p, an FPV simulator that does JUST that.

More, this game runs on an open source engine called SimITL, it's the thing that implements the whole quad running Betaflight on it. And it has good physics since it emulates the actual PID behavior of an actual quad (and all other things flight controller does) so you can experience a real prop wash, for example.

Seeing these things a couple of months ago inspired me, but I did nothing with it at the time.

...

Did you witness the huge wave of people modding games with AIs? I surely did.

The whole thing reminded me about something that people were doing years ago with Super Mario 64 decompilation, I'm talking about libsm64, and if you don't know what it is, it's the thing that lets you inject playable Mario from this game to any other game ever.

Now put these two facts together: there's the FPV community that wants to mod games to put FPV drones there, but it has to be done from scratch for each game ; there's libsm64 that lets you play as Mario with the EXACT SAME physics, animations, controls and behavior as in the original game across all the different games.

We don't develop Mario from scratch each time we do a mod with libsm64 do we?

Why developing an FPV drone each time then?

I knew what I had to do. I had to create libfpv, a spiritual descendant of libsm64 which integrates FPV drones in the games the same way libsm64 does it with Mario.

Implementation

I knew I would be too slow doing all this manually, so I let AI do all the technical stuff (it's unfortunate to say knowing that some years ago I've been doing some C/C++ projects manually and rejected AI, as well as developing this site you're currently reading this article on, and compiling Linux kernel with HDMI CEC drivers for my work, manipulating the offset bits to let the driver access peripherals, writing this to device tree so kernel remembers about it, and AI being unusable in there because it just knows/understands nothing... Yet today I have to reconsider this and start using it seriously). It's just what software development is today.

AI was used even at the technical conception level. I've explained the idea to Claude, told it that I wanted libfpv to be based upon SimITL and made it ask me technical precisions (like where would the Betaflight logic run and whether it's the game or libfpv that would calculate drone physics) and get me a prompt with full specifications to give to Claude Code to initialize the libfpv project.

Claude Code (Opus 5.5) started building the project with submodules such as SimITL, Betaflight itself, Dear ImGui... And then compiling a standalone example which kinda looked like pr0p with worse graphics and boring map. But that was enough for me to understand this thing has potential.

I didn't even discuss contents of the ImGui menu, it just understood by itself that it would be used for radio bindings, and even drone configs themselves as if Betaflight Configurator wasn't accessible (but it was thanks to SimITL; not the case with browser builds tho).

While I was early in development I made libfpv compatible with any OS : Windows, Linux, macOS. Probably even consoles but not officially yet (I have a PS Vita so lmk if you want me to do something with it).

I also worked on polishing the Betaflight integration. SimITL was doing it but didn't implement some things like GPS elements on the OSD and the working turtle mode, also handling the OSD disable / airmode switches... I can now do my own perfectly "realistic" FPV simulator, that can be integrated wherever I want!

After two hours of running Claude, once the initial version of libfpv was done, I made it try integrating libfpv into Super Mario 64. That's funny because the project itself comes from the idea of libsm64. After all, that's a game of my childhood.

I made it working on sm64ex fork so the first integration wouldn't inject into the game but rather be integrated in its source code directly.

I left Claude working for... 30 minutes. In 30 minutes it could provide an MVP that was tested already.

It made the drone interactive with the environment, working collisions and anything else related to the sm64ex <-> libfpv link.

The result was so good I had to post this on Reddit. Then I went sleeping. It was late. I had to do maths the next morning.

Seeing that people love the idea, right the next evening I wanted to test out the real injection into a game, a big game. Among all the games that were interesting, I chose GTA IV. Not GTA V because its existing FPV mod is fine, and not GTA San Andreas that I really wanted because I just didn't have it installed.

I left Claude working again but this time it was harder: first of all you have to prepare modding tools, and after an hour of waiting you have an initial version with a lot of bugs. I debug it, make my agent fix things... In 3 hours overall, I had another libfpv integration working. So I had to post this on Reddit too!

If I spent 3 painful hours of debugging, can we still call it vibe coding? 🤔

According to Wikipedia, yes, knowing that I only tested the output code and guided AI into fixing it, and didn't review it entirely. What a shame 🙈

Conclusion

After GTA IV I finally understood that libfpv has a future and I have to ensure this is the future we need. Instead of trying other games I started enhancing the libfpv core itself and reworking the physics because I had some feedback on that. All that to make libfpv (and its integration into SM64) feel less floaty and sloppy.

The integrations I wanna do later are of course GTA San Andreas and BeamNG.drive, the games I mentioned already. There'll be even more, I'm sure.

There's also another secret project coming with libfpv, you'll see 😉