The Context:

Around 4 years ago, a site called poe.ninja was created, which provided the Path of Exile community with a centralized place to look up common items that were on the market in the game.

In Path of Exile, there is no centralized auction house in which players can exchange items for currency, so it results in individuals putting up listings for their items which end up exposed through the Public Stash Tab api, where sites such as poe.trade or pathofexile.com/trade can consume to provide an avenue for players to search for the items they need. poe.ninja would also consume this data, and provide a fairly accurate estimate of the going rate for different currencies, and other commodities that are traded often. The site also includes stack overflow style datadumps in csv format of ratios at the end of each day, for almost 4 years while the game exploded in popularity, about 600mb worth of csv files as the items poe.ninja was tracking grew.

The Idea:

I wanted to create an api that can be consumed that will parse through this data, and those who wanted to find out the value of a certain currency at a certain point in the game 2 years ago, they would be able to find it. It would also perform some simple tasks such as finding the created spike fall in price from day to day, or the difference in ratio from one point in an event to another

I also wanted to practice API creation in node.js, using everything in the MEAN stack except for Angular, as it's an api and does not need a frontend.

Postmortem:

Express in node.js makes it dirt easy to create a api, especially one that required not very much business logic as mine. Mongoose as an ORM/ODM was a little different to LINQ, but after a little bit of tinkering it also can do everything I needed. Using express-validate also made making a middleware to do some basic input validation is fairly simple as well, especially when i discovered that you can extract a lot of functionality so it doesn't over-complicate my routes file.

The longest part of this project was tinkering with how to put 600mb worth of csv data into mongodb. I initially tried to compile all the data into two files, one for each collection. However, my pc quickly ran out of memory, so I had to split the file into 6 or so files and import them individually.

Given that new data is being added every event in the game, creating a bash script to automate this may be worth a revisit in the future. Also, more endpoints should be added when I figure out which ones are needed. This could also be used to explore different rate limiting middlewares in node.js. Node is known for being super easy to prototype apps like this so it should be super easy to do so.

Go to the Postman Docs