Rachel's Yard

| A New Continuation
Tags Scraping
Oct 15 2016

What the Hell Is This?

It is basically a glorified calendar web app written in VueJS (no backend) that basically helps you (students) plan/search your classes better.

Here's a TL; DR page for you.

What the Hell? Why Reinvent the Wheel?

Well, sometimes the school's AIS is too slow for my likeky. Also, I have always dreamed of being able to enroll classes with ease. However, a typical enrollment process/checking for classes has always been like this:

  1. Search your classes on the AIS (Now they have a better interface)
  2. Add the classes that you want to the shopping cart
  3. Oh wait, before you can do that, select the section that you want
  4. Then you add your class+section to the shopping cart
  5. Add classes * click *
  6. Ah shit, class conflicts
  7. Repeats

Finding your classes should not be that hard.

OK, So It's a Calendar App, No Big Deal, Right?

You are wrong.

For starters, where are you getting the data? It was sort of impossible before, until the school rolled out a better interface on PISA, where it uses bootstrap in 2015, and it is actually human readable. Now we can use all kind of crazy DOM parser to find the class data.

Of course, me being me, always write spaghetti code, then fix later, this is how it looks like right now:

split = sectionDom[i].children[1].children[0].data.split(' ');
section.num = split[0].match(/\d+/g)[0];
section.sec = split[2];
section.loct = [
t: classDataCompatibleTime,
loc: sectionDom[i].children[7].children[0].data.replace('Loc: ', '')

section.ins = sectionDom[i].children[5].children[0].data.trim();
section.cap = sectionDom[i].children[9].children[0].data.substring(sectionDom[i].children[9].children[0].data.lastIndexOf('/') + 1).trim();

Well, don't worry about it, it gets the job done, at least for now. I will use prev() and next() and what not when I actually have time to improve the code base.

Fine, So What? Does Your App Enroll Classes For Users Too? HACKS?!

No, it does not enroll users automatically.

Well no shit sherlock. It involves student credentials, and I don't want to fuck with that.


Calm down, it will notify you when your classes are opened. Basically, I have a dispatcher and a bunch of workers to poll data from the website, and insert the changes to the database. It does all sort of magical stuff in the background. Allow me to explain:

Architecture of SlugSurvival

  1. The Data Fetcher will periodically compare the term list on S3 and PISA (usually every 4 days or 7 days). If there are new terms available, then it will fetch the newest term automatically, and upload the course data to S3. If there are no new terms, it will refresh the data for current quarter, until the drop deadline since there are usually changes to the courses up until drop deadline.
  2. The Data Fetcher will also spawns workers periodically to fetch data on RateMyProfessors (usually every 14 days), and only doing so incrementally.
  3. Then, the frondend loads the data from S3, and you will see something like this: http://url.sc/fall2016
  4. The real MVP here is the watcher. It will poll openings data from PISA and insert the changes to RethinkDB, and db2Queue will have a changefeed to push the delta to another queue, where the notification API can notify the students about their class openings.
  5. Of course, when you have time series data, you should graph them
  6. At the time of writing, I'm still trying to improve the automation aspects of the notification component (Tracked Here). But the idea is to unsubscribe users automatically after drop deadline and etc.

So yeah, this is sort of a big project in terms of reliability and automation requirements. I do want to talk to the school and see if they want to use this as part of the AIS.

I will update this post when I have more time and more changes made

Weightless Theme
Rocking Basscss