Skip to content

edelkas/nscrap

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

34 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Scrapes scores from the N v1.4 server into a MySQL database. Each score contains
the replay id, the actual score in frames (at 40 fps), the player id pointing
to a table of players, the rank, the episode number, the level number, and the
demo (5 ones in the case of episodes).

USAGE:
  * 1) Configure the script by modifying the values of the constants defined at the
       start of the code. The comments explain their meaning.
  * 2) Manually create a MySQL database named 'n'. Alternatively, you can modify
       the CONFIG constant which contains the database parameters.
  * 3) Run the script. There are various commands available, which can be passed
       as command line arguments or inserted after execution. Type "help", or
       any incorrect command really, to see the list of commands.

To have a fully correct database I recommend the following steps:
  1) Execute 'scrape' to scrape the complete database.
  2) Execute 'diagnose' to find errors in the database.
    2.1) Execute 'sanitize' if there are hacked scores.
    2.2) Execute 'patch' if there are corrupt or missing scores.
    2.3) Execute 'uniq' if there are duplicate scores.
  3) Execute 'seed' to fill in the rank field.

NOTES:
  * The first execution will take longer to start up since the database needs
    to be set up.
  * Some scores do not exist, so the total amount of scraped scores will be
    lower than the total number of different ids.
  * A few scores do not have demos, so the final number of entries in the
    'scores' and 'demos' tables may not coincide or some demos maybe be null.

NOTE ON COMPRESSION:
  For the purpose of compression, the demo code is transformed in a way that
  exploits their structure to obtain an almost tenfold compression. To recover
  the original demos read the following details.

  Structure of a demo: The frame count, followed by a colon, followed by a
  series of numbers separated by vertical bars. Each number encodes 7 frames,
  and on each frame any combination of the 4 possible inputs can take place
  (left, right, jump, or hold), with the exception that jump and hold cannot
  occur simultaneously. Therefore, there are 12 possible combinations for each
  frame and thus 12^7 = 16M different frame codes.

  To compress the demo we transform this code into a string with each character
  representing a frame input, with only 12 possible different values. Because
  of the nature of a demo this is not random and is in fact highly compressible,
  so we deflate it using Zlib, and then to obtain an ASCII representation we
  encode it in Base64.

  To see the details of how to encode and decode each demo chunk into the string
  of inputs for each frame, see the functions "demo_encode" and "demo_decode"
  on the source code.
      
STATS:
  After using the program to scrape the whole database, and cleaning it, I
  present some figures.
  
  * 365130 scores.
  * 22222 players (that's right!).
  * 2189 duplicated/triplicated scores.
  * 22 scores without a valid demo.
  * 9 duplicated players.

About

Scrap N server to backup the data

Resources

Stars

0 stars

Watchers

1 watching

Forks

Releases

No releases published

Packages

 
 
 

Contributors

Languages