Is GPT coming for my job?

The new possibilities offered by GPT are a breakthrough. What does it mean for software engineers?

Mar 1

Artificial Intelligence is popping up everywhere. People are playing around with the betas of ChatGPT and Bing AI. The results are exciting, puzzling, frightening, and hilarious.

And many software engineers are asking themselves “is my job at risk?”

The answer is not simple. Why? Because there are more questions embedded within this one. Here are some of the ones I am thinking about:

Will AI replace software engineers?
Will products like ChatGPT replace software?
What should software engineers invest in now?

The answers to the first and second questions are “no” and “maybe.” Let’s dig into these two questions to help us answer the third.

Will AI replace software engineers?

“AI is getting so good at writing code, we won’t have jobs in the future!”

Sentiments like these were common as early as 5 years ago. A fear that we are simply training the model that will one day replace us. It is not an unfounded fear.

Then GitHub Copoliot arrives. I got on the beta list and gained access. With a copy of Visual Studio Code open and an empty file ready to go, I type in this comment:

// Pokemon Go clone goes here

Nothing happens. Of course, my expectations are not actually this high, but it never hurts to try!

I then start using it in some existing projects as well as some blank files. If you are new to a specific language or framework, Copilot is a great tool. It makes it easy to get personalized examples of code you could reasonably use. I can see Copilot becoming an alternative to coding tutorials.

Copilot also shines at helping write unit tests. Unit testing libraries typically ask you to describe the test you are writing. Copilot can read other tests from your project, then write a new test for you based on your description. It will even suggest functions and properties for you to add to the code you are testing. This is gold if you want to do Test Driven Development.

Beyond these two areas, Copilot is less useful. Copilot trains on open-source software, which has a variety of licenses. You may unknowingly use a piece of open-source code in your project. Worse, if you treat the code as your own and do not give proper attribution, you are violating the license.

It is possible to opt-out of having Copilot train on your open source projects. Many software engineers have done this. But afterwards, some engineers discover Copilot still offers their work verbatim.

You should keep the caveats above in mind as you use Copilot and similar tools. But don’t let that stop you from trying it. Think of it like adaptive cruise control in cars. It can help automate some of the more repetitive tasks, but you still have to steer. Copilot can yield some surprisingly good results. But don’t expect it to replace too many software engineering jobs.

Will products like ChatGPT and Bing AI replace software?

Artificial intelligence may not be replacing software engineers themselves in the short term. But will it replace software? Maybe.

Rather than ask ChatGPT to write a Pokemon Go clone for me, I decided to try solving a specific problem. I asked ChatGPT “how much would it cost to run all buses with 10 minute headways in Grand Rapids, Michigan?” The first responses I got from this line of questioning were hedged. It suggested contacting “the Grand Rapids transit agency” for more budget details.

I asked about specific lines in the system. Then I asked about a rough estimate. I was still not getting anywhere. I then asked if it could use a “generic model” to provide an estimate. This finally started getting us towards an estimate:

“According to the National Transit Database, the average operating cost per hour for a standard transit bus in the United States is around $200 to $400.”

I then asked it to do some basic math with parameters:

“If the line is 10 miles long and we want to run it at 10 minute headways for 14 hours a day, what would be the rough estimate for the annual cost?”

OpenAI finally arrived at an answer of $1,533,000 and also showed how it got here:

Picked the mean value of $300 for the hourly operating cost
Multiplied this times 14 hours: $4200 a day
Multiplied that times 365: $1,533,000

But this information was for a single bus. Even though it repeated back my 10 mile figure in the explanation, the line length was not considered. It also repeated that I should contact the transit agency for better information.

At this point, I felt that ChatGPT was not going to substantially improve its answers for this chat. I could have trained it by asking things like “how many buses will we need?” But I probably would have had to coax it into coming up with the average speed of a bus line. I may have even had to supply that information myself.

Since the logic wasn’t going anywhere further, I asked it to turn everything into a Python function. It obliged:

def estimate_annual_cost(bus_line_length, operating_hours_per_day, avg_operating_cost_per_hour):
    daily_operating_cost = operating_hours_per_day * avg_operating_cost_per_hour
    annual_operating_cost = daily_operating_cost * 365
    return annual_operating_cost

Notice there is an argument for bus_line_length, but it isn’t used! I wondered if making the headway a variable would make the function make more sense. So I asked ChatGPT to do that:

def estimate_annual_cost(bus_line_length, operating_hours_per_day, avg_operating_cost_per_hour, headway_minutes):
    daily_trips = (1440 / headway_minutes) * operating_hours_per_day
    daily_operating_cost = (daily_trips * bus_line_length * 2) * avg_operating_cost_per_hour
    annual_operating_cost = daily_operating_cost * 365
    return annual_operating_cost

This result is much more plausible. It takes bus_line_length into account. It also calculates the number of trips. It even realizes that we need the line going in both directions, so it multiplies everything by 2.

It then insisted that if you ran estimate_annual_cost(10, 14, 300, 10), you would get 1,533,000 as the result. So I ran the code locally outside of ChatGPT:

4415040000.0


** Process exited - Return Code: 0 **
Press Enter to exit terminal

With billions of dollars on the line, you should definitely fact check anything coming out of GPT! Still, I am impressed with how far ChatGPT got in solving the problem once I convinced it to use generic data. You might not be able to use the results verbatim. But you can use results like these to help you come up with ways to approach problems.

What should software engineers invest in now?

AI may not be coming directly for your job. It also may not replace the bulk of software in existence. (At least, not in the short-term.) But does that mean “business as usual” investor-engineers like us? Absolutely not!

With AI becoming generally available, we will have more “users.” Many of these users will be machines. Keeping our software understandable, clutter-free, and usable will be paramount for AI. Maintaining accessibility is already crucial for ensuring everyone can use our software. Going forward, “everyone” will also include machines.

Collecting and assembling clean sets of data will be the key for using AI effectively. The results of AI will only ever be as good as the data it has access to. When it does not have access to good data, its models approximate the response that is most likely to be correct.

Current chat-based UIs generate responses immediately, even when they are demonstrably wrong. The next generation of AI UIs will not need to impress us with improv skills. They will likely provide a workspace for defining a reusable model based on data sources. Then the results can be evaluated and refined over time.

Even with better tooling and good data, you are still going to need to review the outputs from AI. Software engineers already review each other’s code before merging it in. Similarly, reviewing the work of AI will be essential. Regardless of whether the output is code or more data, omissions and errors will need to be caught.

Lessons from the past

Before I wrap up this post, I will tell more story. Let’s travel back to the 1970s. An exciting new tool is introduced to the world of computing. One replacing the need for computer programmers. So simple and so powerful, it promises CEOs will be able to use it every day to make critical business decisions.

The name of this tool is SQL. It queries tables and rows in a relational database. It can crunch numbers, summarize data, and provide up to the minute results.

As we come back to the present day, SQL and relational databases are everywhere. They are not only every bit as powerful as they were in the ‘70s, they are even more so. Many contenders have tried to unseat them: document databases, key-value stores, and blockchains. But they are still here and do not show any signs of leaving us anytime soon.

The current prevalence of SQL is hard to overstate. Just one thing: CEOs are rarely using SQL on a daily basis, if they even know what it is. Even software engineers tend to not touch SQL directly. Instead, SQL statements are generated commodities. They are as likely to deliver a revenue report as they are to show how many people liked your social media post.

When breakthrough technology arrives, many software engineers worry about their careers. Then as the technology settles in, it turns out that there is just as much demand as ever before. But the expectations are raised. We don’t settle for what we could do yesterday, we invest in what we know will be possible tomorrow.

Don’t fear artificial intelligence. But do not stay still either. Look for the investments you can make today to use AI to your advantage. Your future has never been more exciting!

Code Like An Investor