A World of Warcraft character and statistics tracking website
*NOTICE* This site is currently being activley worked on atm. please stand by ...
I have implemented a results plugin system so that after the filters have done there work the results get handed off to a series of result plugins that then present those results in meaningful ways.
One of which has been a statistics plugin I've been toying with. Screenshots below are a 24 hour snapshot from the Sisters of Elune realm.
Some screenshot using multiple search filters.
There is no point in capturing data of you can't data mine it right?
The screen shots below are of the data mining application that I am developing. I've had it running for the last few weeks but recently I had to gut alot of the code to simplify it. I am using alot of PLINQ, and Tasks to make it as multithreaded as possible but my initial aproach was causing some serious headaches with cross threading issues and collections being modified. These last few days I have simplified the code base down and made it easier to manage.
But there is still alot of work to do before I release the source code to both the data mining app and the data scheduler app.
As part of my data capture tool set I wrote a scheduling app that downloads auction house data at set times. The app also supports plugins and I have plugins that compress the auction house .json files into .zip files, another plugin that scans the auction house data for any newly discovered item id's and queue's then up for download. I have been using it to capture data since december 2013 and have aquired over 1000+ auction house captures consisting of 1.5Gb of compressed .zip files. I have provided some screen shots below.
File storage for 10 high population realms, using 7-Zip LZMA2 Ultra compression, over the course of one year would require roughly 130Gb of storage. Each individual realm requiring about 36Gb of storage per year.
I don't have time to work on this site at the moment and don't see myself having time for the foreseeable future. Perhaps when Blizzards Titan is released with a armory api i will re-purpose the site for that game but for now this site will remain in a state of suspended animation.
The goal of this website was to track characters and guilds, and allow say a raiding guild recruiter to search for players with desirable data. All plotted out in graphs and statistics. So you could also follow a player or guild and track there leveling progress, achievement, and statistics progression visually. It was an ambitious project one I hope to come back and finish some day.
Below is a few to do notes that I have kept
When posting items to the auction house you have the option to list the items for 12, 24, or 48 hours. When I post items to the auction house I post all my items at once and post them for 24 hours at a time usually at around 10pm. It can take me up to 15-30 minutes to post 200-300+ items. This allows me to setup a schedule and a known time for when I have to log into the game and re-post any items that did not sell.
I try to stay away from posting items for 48 hours because the market can often fluctuate and you may find that posting an item on Monday for 48 hours at 100g that item may only be worth 50g by Wednesday and many other people will more then likely have under cut you. I post all my items for 24 hour periods because I can log in once a day and re-post any item that did not sell on a fixed schedule.
I try and stay away from posting items for 12 hour periods because it would require me to spend another 15-30 minute session re-posting items. You have to balance out how much time you are willing to spend managing the items you have on the auction house verses how much time you plan on actually playing the game or raiding etc.
Posting an item for 12 hours is good for when you want to undercut your competition and almost always have your item listed as the cheapest item available. But it requires that you re-post the items every 12 hours or twice a day.
The idea here is to ensure that you always have items on the auction house for sale at all times. If you schedule your posts every 12 hours at say 8am Monday morning but you forget to log in and re-post the items later at 8pm Monday night, you will probably not get around to re-posting those items again until 8am Tuesday morning. That means that your items were not listed on the auction house for a full 12 hours resulting in a potential loss of sales during that time.
Another trick to try and price bump an item on the auction house over time is to post multiple items of the same type using different characters.
Each character undercuts the previous character by a slight margin. To the purchaser it will appear as though a number of people are compeating on the auction house but in reality all purchases of that item will be buying it from you!
This will be the first of many posts relating to how to make gold using the auction house.
Currently on the Proudmoore realm Cocoa Beans sell for about 2g each. You will typically get 2 stacks of beans from the imported supplies, so I make about 80g for the cost of one Chef's Award.
Depending on AH prices you can also try and purchase Crate of Tasty Meat for x2 Chef's Award and sell the meat.
Generally you will want to stick with just getting Cocoa Beans because when you factor in the number of Chef's Awards and meat items Cocoa Beans will still be more profitable.
The site has been on hold for some time now and I keep meaning to get some work done on it but can't seem to make the time.
I am planning on creating a new section just for tracking the auction house items over time etc for the various realms but again I need to make time in order to get it done.
In the mean time I have been thinking bout doing a periotic blog posts about how to make gold with the auction house. At least then there will be some new content being added to the site.
Another auction house tip is to keep an eye on price history for an item. If an item was selling for 100g last week but has slowly gone down in price to say 50g, but there is no more of that item on the auction house. Try re-posting the item on the auction house for 125g. If anyone posts the same item they will generally undercut you at around 123g. Which is still more then what the item was selling for a week ago at 100g.
Repeatedly do this and try and figure out what the maximum people are willing to pay for that item before they attempt to aquire it by other means.
So I was just surfing the net looking for some sample code and remembered that I had not checked to see if Blizzard had there new armory API availible. Turns out they just made it availible recently.
So it looks like I will be doing some more work on this site in the coming weeks. Wuhoo!
I have updated parts of this site, but have removed access to the armory data for now.
The armory data is over a year old anyway, and I am still in the process of making changes to this site.
Have not made any updates to the site since last post. Still waiting for blizzard to provide xml data feeds for armory characters. There has been no way to get access to the xml feeds since they updated the armory to the new website format and thay have been taking there time getting the new feeds up and running.
The only other way to get the data out of the armory is to download the website and parse the HTML and I am not preparsed to go through and recode my tools for getting armory data.
So until blizzard starts providing xml feeds again this site will have to be put on hold. grrr
I have returned from the great north woods, but have not had much time to work on this site much. I've been busy with other things like XNA 4 and windows phone 7 type stuff.
I have been waiting for my hosting provider to add asp.net 4.0 support but it may be a wee bit before that happens so the new site is still waiting to be set free. Also I have not been downloading character stats from the armory either because it takes a very long time and I am planning on changing my approach to the site.
As it is up until now my approach has been to download the entire Sisters of Elune character roster once a week. About 40,000+ characters give or take. But this introduces problems with the amount of storage required to store that data as well as how to efficiently store it for quick retrieval. I have a working theory that may work better and that looks good on paper but have not implemented/tested it yet.
Instead I have decided not to download the whole SoE character roster every week and finish work on my Wowtracks client downloader application. This application will allow registered users of the site to track any number of characters they wish to and thus the completeness of the data on the site will be the sole responsibility of the community of users that wish to participate.
So if only fifty people use the site and those people only track say 1500+ characters across the various realms, then the site will only have data on those 1500+ characters and no others. If a person visits the site and finds that a character on a particular realm is not being tracked they can choose to register a free account with the site then they can create a list of characters they are interested in tracking. After that they just run the client downloader app and leave it run in the background.
The downloader app will download the users list of characters they are tracking first and when it's done it will attempt to download characters from other users tracking lists, until there are no more characters left to track. Characters will be restricted to only being downloaded by the downloader app once a day. This is by design because multiple users could be tracking the same toons and downloading a toon twice or more in a 24hr time span is some what pointless.
This makes the site only as good as it's community participation. But like I say if some one wants to participate and donate there time to track 100,000+ characters and let the downloader app run endlessly they can do that to. Even if they have no interest in tracking all those characters but want to help the site anyway.
I'm going to be heading up north to go mushroom picking until the end of october so, no internet and no electricity fer me until then. Also means no updates to this site until then either.
Have not made any posts latley but I'm still pluggin away at the new asp.net mvc site. It's comming along but again I am faced with a difficult decision on how I can store the character data effeciently as well as how to pre-calculate the character, guild and realm statistics. grrrr.
Anyway just posting to let u know i'm still on the job :D
I was also looking into uploading the new version of the site I'm working on as from what I'm reading I can compile it for .net 3.5 but the problem is that I am using features found in .net 4.0 such as the C# dynamic keyword as well as Tasks so the only way I could get the new site running would be to rewrite those pieces of code so they would be compatable with the .net 3.5 framework. But again the whole reason for using .net 4.0 is because the features allows me to write alot less code.
So that is something I'm going to have to think about.
Certified genius right here people teehee.
*takes big sigh of relief*
Finnally finished all the talent trees for each of the ten classes. I was doing it manually by hand editing the structure and values but stoped for a few days to think about doing it a better way. Had I continued it would have taken me weeks of painstaking editing and verifying.
So I did what any good engineer or programmer would do and build a system that did it for me :P It actually did not take as long as I thought it would to code. The little utility application that I used allowed me to define each talent tree layout, and max points per talent item, and icons etc.
I got the utility app working and all ten talent tree's done in a single day!
One thing I have been thinking about though is that wowtracks is aimed at being a statistics and character tracking website, so I have decided not to have any talent tree building functionality. If you want to playaround with talent trees I provide links on the talent pages to either the wowarmory talent builder or wowheads talent tree builder.
Dang it! I was just building the talent tree page for hunters when I just realized that I'm not capturing talent data for pets. So hopefully this will be the last week that there will be missing pet talent data.
I've already started the armory crawl this week and it's 50% done already so hopefully by next weeks armory crawl pet talents will start being captured.
*Sigh* Well I setup a framework for displaying talent tree information. Here is a screenshot of the first warlock talent tree work in progress. Only 29 more talent trees to go X.X gah! This is gonna be a painstaking process QQ
Still working on the asp.net 4.0 mvc version of this site, and came across a bug. The bug has to do with how I was interpreting the equipped items that a character has, specifically the enchant that the item may have. It was recording the wrong enchant id, but now it's fixed.
Also got the basic framework complete for generating precalculated guild/realm/character data. This allows me to precalculate and store statistic data rather then doing it on the fly which is much slower because a number of statistics require that I look back into the previous weeks captured data set and compare it with the current data set for the week.
I know I keep talking about these fixes, and features, but have not added them to the site yet. I'm still waiting for my hosting provider to support asp.net 4.0. Once that hapends you will be able to see what I have been working on. But until then as I mentioned in a previous post, updates to this site as is will be minimal.
How much do I hate web development? This animated gif pretty much sums it up nicley.
Sigh* If it's not one thing it's another. The most simplest of things seem to turn into mega projects to get done. I've been coding the site anew in asp.net mvc. It is so clean, neat and relatively straight forwards to use. I like it a lot. But it is still new to me and there are annoying learning pains.
Some how my 26k+ list of character names is getting contaminated by a few random names that are gobblely gook. Names that should not be getting through the system and into the character list but they are. It does not affect the site in any way. Any attempt to grab armory data for the garbled character name from the armory naturally returns a no character found result, thus the character is not added to this site.
So there is still a lot of bug checking and validation needing to be done. *Takes deep breath, pushes up sleeves, and continues with the torturous act of coding.
I may not be making much more updates to the site from now on, or rather not as frequently. I am in the process of learning asp.net mvc 2 and it is way cleaner and easier to work with. I am already well on my way to re tooling the site to run under asp.net mvc so the next few weeks there will not be to many updates til then.
I will continue to download armory data for SoE every week and upload it to the site so new data can be viewed. I am also near compleation of a distributed armory downloader. With this distributed downloader I can install it on any number of client machines and capture armory data faster. I have already put it to use it to capture data for May 25, 2010.