First impressions of the GitHub Copilot preview

For a long time now, some have predicted the death of computer programming, but it never happened for various reasons, the most important being that programming is just as much an art as it is a science or an engineering discipline. Touted as an AI assistant for programming, GitHub’s Copilot, currently in limited-access technical preview, sought to automate writing code a step further than IntelliSense and equivalent tools allow. It is not entirely autonomous. You have to declare your intentions to it before Copilot can generate meaningful code, and you also have to supervise it to get it back on track when, inevitably, it goes off the rails.

Copilot is a cloud service with interfaces for Visual Studio Code (installed on the developer’s machine or in the cloud on Github Codespaces), for a Jetbrain development environment like IntelliJ IDEA, and for Neovim. The cloud service is a code prediction engine powered by OpenAI Codex, an artificial intelligence model trained on billions of lines of public code. Yes, both Codex and Copilot have been controversial. Before exposing potential copyright and privacy violations committed by Copilot (as complained by the Free Software Foundation), it should be understood that Codex was trained on publicly available code in a manner often considered fair. within the machine learning community. It should also be understood that Codex is a code synthesizer, not a search engine. But the developers of Copilot themselves acknowledge that he still has a lot to say on the matter: “…this is a new space and we want to engage in a discussion with the developers on these topics and lead the industry to establish appropriate standards for training AI models”.

How Copilot works

Because OpenAI was trained on publicly accessible source code and natural language, it understands both programming and human languages, says GitHub. “Copilot’s editor extension sends your comments and code to the GitHub Copilot service, which then uses OpenAI Codex to synthesize and suggest individual lines and entire functions.” And the service uses user choices to improve future suggestions.

Copilot being in limited preview, you must first register on a waiting list before you can access it. Once the welcome email has been received, you can go to the GitHub Copilot extension page on the Visual Studio Code marketplace to install it. Then you have to authorize the extension in Visual Studio Code. The start page contains a tutorial that can be followed, starting from point number 2. In this tutorial, we create a .JS file of the type “function calculateDaysBetweenDates(begin, end) {” and we obtain a function fully implemented, inferred from the name of the function. On the following tutorial, same page, one types a comment summarizing what a function should do and one gets a fully implemented function inferred from the comment, even if the function name is too general to be useful.

The GitHub Copilot extension page on the Visual Studio Code Marketplace. As can be seen at the top of the screen, the extension is already installed. (enlarge image)

Screenshot of Visual Studio Code with Copilot active (enlarge). We see the 1st tutorial in the documentation with the suggestion of “ghost” code below the entry, as well as the pop-up control bar of Copilot. (Credit: Martin Heller/InfoWorld)

In addition to deducing the function bodies from the name of the function and a summary comment, Copilot can draw inspiration from other codes in the file that is being edited as well as from the names of variables. For example, if you enter a colon on the keyboard after a variable name in TypeScript, Copilot tries to fill in the type. If you enter “var test1=” on the keyboard, Copilot locates the word “test” and generates an executable test for the previous function. If I enter several lines that form a repeating pattern, Copilot will try to generate other examples on the same pattern.

Copilot can be used with a wide range of frameworks and languages, but it works best with Python, JavaScript, TypeScript, Ruby and Go. More recently with Java, the C language family (C, C++ and C#) being planned later. Some users have mentioned that it works well with some popular JavaScript frameworks like React. In the future, GitHub expects Copilot to consider more of the project’s code in context, beyond just the current file.

(larger image) In this example, lines 8 and 9 were generated by typing the start of the lines and a series of tabs. Line 10 was entered, along with the start of line 11, and Copilot completed line 11. In TypeScript, after compiling to JavaScript run under Node.js as can be seen at the bottom of the screen. Note the incorrect comments generated on the expected result values ​​in lines 8 and 9.

The shortcomings of Copilot

First of all, Copilot does not always generate good or correct code. Worse, it sometimes generates code that cannot be run. I encountered these three scenarios in my test. The code that Copilot generates should be absolutely checked, as if it had been written by a novice programmer who knows how to search on Google but needs to be checked closely. LOne of the ways to avoid accepting the first suggestion offered by Copilot is to resort to the Open Copilot option in the context menu, or to use the shortcut Ctrl-Enter to display the Copilot suggestions window in a tab separate. You can then consult the 10 suggested solutions and accept the one that comes closest to what you want. We can then edit the generated code a little to improve its robustness.

GitHub has done a benchmark on Copilot code generation. “We recently compared it to a set of Python functions that have good test coverage in open source repositories,” he explains. We cleared the body of the functions and asked Copilot to fill them. The model did it correctly on the first try in 43% of cases, and after 10 attempts in 57% of cases. And he gets better every time. Obviously, 43% correct is not a very good (or even acceptable) accuracy score for production use, even if it is an impressive achievement for a new code generation technology. Nevertheless, good code reviewers will be able to rectify the code generated by Copilot to make it correct and robust much faster than they would write it themselves from scratch, especially if they are working with a library or framework that they have been using for a short time.

(larger image) The tab on the right shows 10 suggested code snippets for the function body. We can accept the one that comes closest to what we want to obtain.

Some examples

The Copilot home page presents a series of small code generation examples and larger examples accompanied by videos of screen shots in the Copilot gallery. It is likely that the software development team will release more over time. It is worth watching the proposed animations and downloading the videos from the gallery.

(enlarge) A Copilot example for sentiment analysis in Python, following the gallery. Parts of six lines were entered, plus a lot of tabs to accept the code. Several suggestions were rejected, including test phrases that generated negatives for the list of positive phrases. The code didn’t run until the Python Requests package was installed on the machine with pip3. (Credit: Martin Heller/InfoWorld)

Overall, Copilot is quite useful in its current phase of development. Its current performance gives hope that the tool will save time in the future. Whether it will be worth buying the commercial product when it ships is still open. It will depend not only on the evolution of its performance, but also on the own skills of the developer who will use it and on his role.

There are several products that claim to compete with Copilot. The most promising of these seems to be Tabnine, from the Tel Aviv-based publisher of the same name. Tabnine looks like IntelliSense on steroids. He can train on the developer’s corpus of code as well as on open source code. Different other alternatives basically search the relevant code on StackOverflow which may question the methodology. It’s certainly worth trying out Copilot in its own environment and tracking its progress over time.