GPT3 vs Leetcode, do you really need your data analyst?
Challenging the most powerful textual model to solve basic sql tasks for me
Abstract
Leetcode is one of the most dangerous places for every kind of software engineers. You definitely begin to reflect on yourself as a specialist when you find yourself completely stuck on an “Easy” task for 40 minutes of your live for the first time.
Of course, I’m not the best Leetcoder in the world, but what about the most powerful ( debatable ) language model by OpenAI? Сan a neural network solve sql problems at least as well as i do?
To check this, I decided to parse all sql problems from Leetcode, get answers from OpenAI sql request api and visualize statistics.
Parsing all databases problems
To do this we only need:
- 35$ for a premium subscription
- a bunch of python libraries for web scraping
After we’ve implemented function to scrape all links using BeatifullSoup we can get html page from our driver ( using page_source attribute ) and extract all links to dataframe ( we also keep information about task difficulty — it will be useful for OpenAI performance estimation at the end).
This is what the resulting dataframe looks like:
Getting description from tasks
You must be logged in (I did it manually, because sooner or later Lееtcode starts asking captcha) and have a premium subscription for this code to work ( otherwise you will have a completely different html layout )
Okay, right now we can extract a task description from any problems using this function.
For example, you can just try using this on random links from dataframe which we’ve created previously.
Getting answer from OpenAI
Right now we have description which we can submit to OpenAI API.
That’s just a visual explanation of how you can get a solution using task description manually.
Don’t forget to add SELECT at the end to give the neural network a hint of what it’s all about
You can also do it via Python with a simple function. We will not discuss the meaning of parameters that can be passed to the model, let’s keep it a field for experiment.
Submitting task and automatic checking for status
Great, we have learned how to get problems descriptions from Leetcode, communicate with OpenAI and get solutions for our tasks.
Next step — trying to submit an answer we’ve received to the Leetcode platform.
I absolutely gave up trying to do this using javascript and regular selenium ( send_keys to a form, ang other stuff ).
However I noticed that pressing the ‘Submit’ button spawns a post request to the endpoint <task_link>/submit, all we need to do is just to change the typed_code parameter and send the solution again.
So, I deсided to use Selenium Wire to get requests from the driver.
I will pull out a brief description from the library documentation:
Selenium Wire extends Selenium’s Python bindings to give you access to the underlying requests made by the browser. You author your code in the same way as you do with Selenium, but you get extra APIs for inspecting requests and responses and making changes to them on the fly.
Let’s check if this is true by writing the following function.
Our last step is to check the result of our submission, nothing interesting by the way.
Business is getting real
Finally, we have all the parts we need for our performance. Let’s go over tasks links and collect answers.
You can find your favourite task in the dataframe below and see if the neural network has nailed it.
Okay, and right now we ended up with 216 questions, 35 bucks spent ( no regrets ), and roughly 3 hours of enjoying the finer points of web scraping.
Let’s look at the data, can already throw off part of the work on the mighty shoulders of an artificial atlas?
Long story short: don’t think so
42 accepted answers might seem not so bad at first sigh, but it is worth looking at the distribution of the tasks by the level of complexity.
The plot is interactive, feel free to click’n’play
In the end, it seems that data analysts still have a chance at the moment, especially when it comes to non-trivial queries.
