Review: GitHub Copilot and C++
03 July 2022
GitHub Copilot is a tool that provides real-time suggestions while developers write code. It offers much more than code completion, a feature found in most modern IDEs. Instead, it tries to predict what the developer wants to write from a couple of lines to entire blocks of code. Internally, GitHub Copilot leverages Machine Learning models trained on an unspecified number of open source projects, hosted by GitHub. Recently, it became publicly available at the price of 10$ a month which spawned some controversy. I have been using Copilot to write mostly C++, so let’s see what the fuss is about.
Controversy
I will start by addressing the elephant in the room.
GitHub Copilot has been around for a while, for free, but only available through the beta preview program.
Whoever wanted to use it, had to apply for it and wait to be selected.
A few days ago, GitHub Copilot became publicly available but with a caveat: You need to pay for it.
This stirred up the open source community who thought that the tool had been taking advantage of their
free work for a profit.
Many saw this move as proof of Microsoft’s “evil” ulterior motives when acquiring GitHub.
To be honest, I believe it was within GitHub’s rights to release a tool trained over data it devotes resources
to host. Was I disappointed? Sure, because I like free stuff. However, talking about betrayal is unfair.
We already know that if a service is free then you are the product. So why become sentimental about it?
GitHub’s contribution to open source is undoubtedly huge, so let’s be content about having GitHub in the first place
and let the community decide which service has the best offering.
The good
I have been writing code on VS Code with the GitHub Copilot plugin for a variety of projects, from large to small, mostly in C++.
The domain is primarily embedded systems, from microcontrollers to middleware running in some kind of embedded Linux device.
GitHub Copilot’s suggestions are impressive and orders of magnitude better than the competition.
Compared to tabnine Copilot’s performance and ease of getting started are nowhere near.
Using Copilot feels almost like “magic” or even “cheating” at times since it allows you to skip writing a lot of code.
I suspect it works even better with other languages and domains, e.g. in web applications.
GitHub Copilot will not help you write better code. Ideally, it will propose the code you were going to write anyway. Nothing more, hopefully, nothing less but this translates to a huge boost in your productivity. This may sound strange to the untrained ear, however software developers know that when writing code, the next line we are about to write is to a good extent “obvious” based on the context and/or some locally recurring pattern. This is where Copilot kicks in, fills the “trivial” parts, lets you focus on the more important ones, and set the overall direction.
GitHub’s tool is great at utilizing context and patterns. The more consistent you are when it comes to following design patterns, naming conventions, and other common idioms, the better the tool will be able to predict what you are going to write. It really shines when you are developing unit tests. After you have written a few test cases, it will either directly suggest the next case or after you write a nice descriptive test case title, as you should anyway, it will basically generate it. You were going to write that going anyway but Copilot made you finish it faster.
A common concern that arises from the usage of GitHub Copilot is whether you will unwittingly copy some open source project’s code.
GitHub Copilot has been trained over a large number of open source projects.
Therefore, it may theoretically suggest a code snippet from an existing project.
Some of these projects could be reserving all copyrights, meaning that you are not allowed to reuse code from them at all.
Others may be published under a copyleft license resulting in your project being potentially considered derivative work
and having to be released under the same license.
How valid are these concerns?
While I am not a lawyer, I firmly believe the risk of infringing someone’s copyright is negligible.
This is because GitHub Copilot’s suggestions are context-specific. In my experience, the snippets it recommends would not make much sense
in other contexts.
Remember, GitHub Copilot will suggest code that you were going to write anyway.
If you are trying to do exactly the same thing as some open source projects do, in the same language and following similar patterns
and conventions then, sure, you may get a piece of code found elsewhere.
The thing is, you were going to write that same code without Copilot.
If a particular piece of code is that obvious, then I have a very hard time how anyone can feasibly claim ownership of it.
The bad
Nothing is perfect and what I described above is the best-case scenario when it comes to the tool’s predictions. While GitHub Copilot can help you write code faster, it may also be distracting or lead you into debugging sessions of code it suggested. While you may be typing some code and are fairly certain what you want to write next, the tool may suggest something completely wrong. Going through the suggestion before discarding it may result in working slower with the tool rather than without it. I do not have data on Copilot’s accuracy. According to my anecdotal and likely very biased experience, its recommendations are around 60% accurate. In practice, you eventually get a feeling of when the tool’s suggestion is wrong and learn to ignore it at a glance. However, when the recommendation seems correct, you may end up going through it to realize that it isn’t, or it needs corrections. This could obstruct your flow and depending on your personality and preferences, it may be detrimental to your focus.
GitHub Copilot’s suggestions are supposed to be context-specific. Unfortunately, from my experience, it does not necessarily take into account what is compilable. In other words, it will occasionally suggest code that tries to use non-existent API calls. You will be surprised, but it will not always be obvious especially if your IDE is not fast enough to highlight the error. You may end up needing to go back after you have completed writing a block of code, to find the culprit and fix it. It would be very helpful if GitHub Copilot would take into consideration the compilation database before giving suggestions.
Another issue is that you may become too reliant on it. I often find myself anticipating its input, even if I certainly know what I need to type next. You may think that the problem is you cannot write code without it, but that is not it. Sometimes GitHub Copilot is simply slow or does not provide any suggestions at all. I have sometimes found myself waiting purely out of laziness and because I am generally intrigued by what it will suggest next, when I could have simply written that damned code and gotten it over with.
Finally, GitHub Copilot may not always play well with other plugins such as IntelliSense. Sometimes, you may just need to inspect a class instance’s public methods but Copilot is not allowing IntelliSense to do that. Instead, it proposes a non-existent method call along with a few other lines of irrelevant gibberish. Ideally, IntelliSense and Copilot should work together to both improve the Copilot’s suggestions and figure out when to yield and let the appropriate tool provide the suggestions.
Verdict
Is GitHub Copilot any good for C++ embedded systems development?
Yes, it is. After more than 6 months of using it I believe it has helped me write code faster, especially code that would otherwise
be boring to write myself.
It has also made writing code more fun. It never ceases to pleasantly surprise me with its suggestions and impress me with the
state of the art in natural language processing and machine learning fields.
Will I pay 10$ per month for GitHub Copilot? No.
Not because I disagree with GitHub’s strategy of monetizing on the open source community or any of that.
I simply do not estimate the value it offers equivalent to 10$ per month.
If Copilot was considerably more accurate and faster, perhaps I could think of it.
Currently, I would only consider paying for it if the subscription fee was at most 2-3$ per month.
To conclude, I will not be renewing my subscription once the remainder of my trial period is over.
It is sad because I like GitHub Copilot, but I cannot justify the asking price.
