THE LAB #9: Scraping OpenSea NFT’s data
Excerpt
Getting winners and losers of the Bored Ape Yacht Club collection transactions
The NFT hype cycle
Welcome back to The Web Scraping Club for a new episode of The Lab. In the past week, a scandal that involves the famous influencer Logan Paul and his crypto project called “Criptozoo” exploded, thanks to the Cofeezilla investigations (you can see the full story here). Basically, it seems that this crypto game has never been delivered for multiple factors but people, trusting the public profile of Logan, put several million USD into it, hoping to have some return, a thing that never happened. It’s nothing new under the Crypto sun, Ponzi schemes promising impossible returns on investments are discovered every day, and surprisingly there’s always someone who got caught in the fishnet.
But this is not a post about scams, only this story reminded me of an ancient word, coming from a far era from the past, called NFT. Do you remember? Funky images with some traits hardcoded in a smart contract, exchangeable like trading cards on marketplaces like OpenSea?
It was the buzzword of 2021, the year well-known for 2 plagues: Covid-19 and the explosion of Crypto gurus that have just left the ICO (another forgotten term) bandwagon, just to hop in the NFT one.
[
Google Trends on NFT
As you can see from this Google Trends chart, the interest is sharply fading away, but during the whole of 2021, the web was full of news and stories about NFTs and new collections launched.
The 2021 of NFT market
One of the most well-known NFT collections is called “Bored Ape Yacht Club”, consisting of ten thousand “Bored Monkeys” images procedurally generated by an algorithm. In May 2021 the launch price for every NFT was 0.08 ETH, approx 200$, and they went sold out in 12 hours. In October they were already exchanged for hundreds of thousands each, quickly becoming a status symbol. They give access to a Discord server where talk with other owners, even celebrities that bought them, and a private section of the collection website.
Yuga Labs, the company behind the BAYC NFTs, in March 2022 raised 450M USD in funding by venture capital firm Andreessen Horowitz, with a company evaluation of 4 Billion USD.
Bored Ape NFTs bought
But when the music stops…
As we have seen many times in different cases when things go up so fast, usually go down even faster. Having a look at the Google trends chart, now the hype is gone and the interest in NFTs, in general, is almost faded away. We can also see it from the OpenSea activity dashboard, where the number of sales for the Bored Apes NFTs went from several hundred per day in early 2021 to a few tenths of the latest months.
To make things worse, the whole crypto economy is passing through a so-called “crypto winter”, with all the major cryptos losing around 70% of their price from their highest peak.
[
Coindesk Bitcoin Chart
I’ve always been fascinated by this world and its dynamics and thanks to the transparency given by the blockchain and the web scraping techniques we’ll see later in this post, I wanted to put down some numbers.
Basically, I’ve tracked down every sale of Bored Ape NFTs that happened on OpenSea from the start to enrich the stats of the collection available on the marketplace.
2 billion USD
That’s the amount of real money moved by The Bored Ape Yacht Club, from May 2021 to the first days of December 2022.
Yes, 2 Billion USD, like the yearly GDP of the whole Central African Republic. Summing up the USD value (at the time of the transaction) of all the sales that happened, you get $2,121,310,433 USD, with an astonishing peak of 283 Million USD traded during only January 2022.
[
Total transactions of Bored Ape Yacht Club
Winners and losers
In a market that had such a hyped start and a sharp decline, who entered first and had the right timing when selling, had a huge advantage.
If we do not consider the first-movers, who got their prize in pre-sale, around ten thousand accounts on Opensea are involved in trading these BAYC NFTs. Around 70% of them lost some money, accumulating a loss of 1.2 Bn USD. On the other hand, the remaining 30% cumulated a gain of 380 Mn USD.
[
Cumulate loss and gains
But if we consider only the accounts that sold more NFTs than bought, if they bought any, they made around 900Mn USD in gains.
As a result, we have a massive shift of wealth (around 1.2 Bn USD) from 7k accounts to another 7k, and half of the latest probably bought the NFTs during pre-sale.
130 vs 127
Another interesting fact that emerges from the transaction data is that 130 accounts made more than 1 Mn USD in trading, while 127 lost more than 1 Mn USD.
One of the top performers is the famous NFT collector Pransky who made alone 5.5 Mn USD by selling 1155 NFTs and buying only one for 1.5 Mn.
With all the buzz and noise around the NFTs, and in particular around The Bored Ape Yacht Club NFTs, it surprises me I haven’t seen these numbers in any article I’ve read. The main reason I suppose is that getting them requires a simple but not effortless job of web scraping, both on OpenSea and on Etherscan to get the USD value of the transaction at that time.
And that’s what we’ll see in the next chapters, the scrapers I’ve created to get this data.
As we said before, the Bored Ape Yacht Club is a collection of 10000 NFTs we can find on Opensea. Each item is numbered from 0 to 9999 and can be found adding its number after the URL https://opensea.io/assets/ethereum/0xbc4ca0eda7647a8ab7c2061c2e118a18a936f13d/.
[
NFT page
The website is protected by Cloudflare, so we need a headful browser to scrape it. We are going to collect all the Sale transactions in the Item Activity section.
[
NFT Transactions
In particular, what matters to us is the account name of the seller and the buyer and the link to the transaction on Etherscan, so we can have more details on the price actually paid for the sale.
I acknowledge a more complete analysis should involve also the transfers between accounts and their re-mapping, but it would be too much effort for this article and doesn’t add too much to the global outcome.
Basically, we’re setting up a python script that, using Playwright, opens a Firefox window for every NFT, reads the table, and saves the sale rows on a CSV file.
We open and close the window for every NFT since I’ve found that this helps avoid being blocked or getting the execution stuck for too much memory used after several requests.
[
Playwright scraper for Opensea
As usual, you can find the code on the GitHub repository for paying readers.
After the execution of the first script ends, we’ll set up a similar one for reading the transaction details from Etherscan.
[
Etherscan page
What’s important here is the estimated value of the transaction on the day of transfer, so we can approximately have the amount of USD people have spent during the last year and a half on these NFTs. Depending on the token used for paying, if ETH or WETH, the price is located in different places on the page so we had to handle these variants. On top, a few links to the transactions scraped from Opensea seemed not correct and I needed to add an exception handling.
Even Etherscan is protected by Cloudflare so again we used Playwright to scrape the transactions that came out from the previous script.
[
Etherscan Playwright scraper
In both cases, Cloudflare was not an issue. Using Playwright with a slow_mo value for the browser was enough to avoid being blocked. Probably a more massive approach with hundreds of thousands of URLs to requests would have been harder to scrape and would have required some proxies.
Putting both the results on a Google spreadsheet and with some formulas and magic, we’ve got all the numbers we need to create an analysis like the ones we’ve seen before.
They could be useful to find NFT collector whales (people like Pransky that collect a great amount of NFTs), traders, or make a sort of due diligence on a collection before entering it.