Friday, March 14, 2014

An instagram recommendation system

I have been very frustrated with the Instagram social mechanisms.... by not knowing how to discover potential instagram users that I am interested in.

Therefore I wrote a little webapp that allows clients to enter the username of their favourite IG user, and the app will recommend related instagram users to the our clients.

For example, if you search justinbieber (Justin Bieber), the system will recommend you Ariana GrandeSelena Gomez, Miley Cyrus and so on. Seems fun, isn't it?

You can try it at http://instammender.com (Instagram + recommender).

Friday, February 21, 2014

Revitalizing old hard drive motors using Arduino

Hard drives use brushless motors (BLDC motors) 

Brushless motors are more durable than ordinary carbon brush motors because they lack a commutator (which is a brush rubbing a pair of cyclic magnets to reverse the direction of current)

For brushless motors they use electronic switches to reverse the current.
In BLDC motors the coils are wrapped on the stator, while the rotor has a permanent magnet attached to it.

However, it should be noted that a brushless motor cannot be driven by ordinary Direct Current.

This article tries to demonstrate how to run a Hard drive scraped BLDC motors by using a microcontroller/ rapid prototyping board Arduino.

This method is based on this video
http://www.youtube.com/watch?v=CMz2DYpos8w

Principles

Use Arduino sends pulses to Darlington Transistors.
There are 3 phases in the motor. Each phase is controlled by one Arduino Pin. 
For example, 

---------time---------------->
phase1pin:100100100
phase2pin:010010010
phase3pin:001001001
Fig 1

where 1 = High signal in Arduino pin, causing Darlington Transistor to supply current to the motor.

Scraping hard drive motors

You need:

Torx screw drivers (especially T8)
common screw drivers

Technique: No special attentions needed. Just unscrew them all. Some screws may be hidden under the label.


(There is a ribbon connected to the drive; DO NOT PULL it off because there are very thin wires which are connected to the coils of the motor inside. I suggest soldering additional wires like the photo above)

Testing which lead is common

Fig 2
Usually a hard disk BLDC is a 3-phase BLDC, which has 3 phases + 1 common = 4 wires

Use a multimeter to test for resistance on those 4 points:

A common lead + coil = 1x ohm
A coil + a coil = 2x ohm (Fig.2)

Then solder extending wires. In Fig. 1, the common wire is in black color.

Connecting to Arduino

Arduino:
- We need 3 digital pins to send the signal, say pin 2,3,4

Breadboard components:
- 1k ohm resistors *3
- TIP122 darlington pairs *3
- diode 1N4004 *3
- battery power 5 to 12V

Connect them as shown in Fig. 3 and Fig 4

Fig 3


Fig 4




Coding Arduino

Simple switching


const int phase1pin = 2;
const int phase2pin = 3;
const int phase3pin = 4;
const int delayTime = 6000; // microsecs

void setup(){
  Serial.begin(9600);
  pinMode(phase1pin, OUTPUT);
  pinMode(phase2pin, OUTPUT);
  pinMode(phase3pin, OUTPUT);
}

void loop(){
  switchStep(1);
  switchStep(2);
  switchStep(3);
}

void switchStep(int stage){
  switch(stage){
    case 1:
      digitalWrite(phase1pin, HIGH);
      digitalWrite(phase2pin, LOW);
      digitalWrite(phase3pin, LOW);
      delayMicroseconds(delayTime);
      break;
    case 2:
      digitalWrite(phase1pin, LOW);
      digitalWrite(phase2pin, HIGH);
      digitalWrite(phase3pin, LOW);
      delayMicroseconds(delayTime);
      break;
    case 3:
      digitalWrite(phase1pin, LOW);
      digitalWrite(phase2pin, LOW);
      digitalWrite(phase3pin, HIGH);
      delayMicroseconds(delayTime);
      break;
  }
}

Improved version

The phases overlap with the previous phase by 1/4 cycle to improve the rotation transition.
In this version the motor is slowly accelerating to the max speed.

Improved switching



const int phase1pin = 2;
const int phase2pin = 3;
const int phase3pin = 4;
float holdTime = 50000; // microsecs
const unsigned long minHoldTime = 1300;

unsigned long p1start,
              p1end,
              p2start,
              p2end,
              p3start,
              p3end;

void setup(){
  Serial.begin(9600);
  pinMode(phase1pin, OUTPUT);
  pinMode(phase2pin, OUTPUT);
  pinMode(phase3pin, OUTPUT);
  p1start = micros();
  digitalWrite(phase1pin, HIGH);
}


void chkP1(){
  unsigned long currentTime = micros();
  unsigned long td = currentTime - p1start;
  unsigned long refractory = 2.25*holdTime;
  if(digitalRead(phase1pin)){
    if(td > holdTime){
      digitalWrite(phase1pin, LOW);
      p1end = currentTime;
    }
  }else if(td > refractory){
    digitalWrite(phase1pin, HIGH);
    p1start = currentTime;
  }
}

void chkP2(){
  unsigned long currentTime = micros();
  unsigned long td = currentTime - p1start;
  if(digitalRead(phase2pin)){
    if(td > 1.75*holdTime || td < 0.75*holdTime){
      digitalWrite(phase2pin, LOW);
      p2end = currentTime;
    }
  }else if(td > 0.75*holdTime && td < 1.75*holdTime){
    digitalWrite(phase2pin, HIGH);
    p2start = currentTime;
  }
}

void chkP3(){
  unsigned long currentTime = micros();
  unsigned long td = currentTime - p1start;
  if(digitalRead(phase3pin)){
    if(td > 0.25*holdTime && p3start < p1start){
      digitalWrite(phase3pin, LOW);
      p3end = currentTime;
    }
  }else if(td > 1.5*holdTime){
    digitalWrite(phase3pin, HIGH);
    p3start = currentTime;
  }
}

void loop(){
  chkP1();
  chkP2();
  chkP3();
  delayMicroseconds(100);
  if(holdTime >= minHoldTime){
    holdTime -= 0.5;
  }
}


That's it.

For my 5400rpm hard disk motor, I used 3.7*3 = 11.1V
The min cycle period = 1.3ms
i.e. it takes about 1.3*2.25*2 = 5.85ms for 1 rev
=> about 10k rpm

Any further decrease in cycle period leads to an halt.

Increasing voltage allows shorter cycle period (prabably due to a stronger torque => higher rotation speed)

Addition info:

I dismantled / disassembled one of the BLDC motor, shown below:
The 4 wires were accidentally broken by me. So I punched the center of the axle to separate the rotor   
              

Please ignore the bearing in the middle. It is totally irrelevant and it belongs to my other projects.

Sunday, March 3, 2013

Implementing reCaptcha in Tornado/Python asynchronously

The reCaptcha client for python lacks a proper documentation.
Indeed, there is no need to implement the reCaptcha-client plugin for python/Tornado. We can simply use the AsyncHTTPClient to directly communicate to the reCaptcha API.

Register a reCaptcha on Google. Get the:
- public key
- private key

The reference I followed is this:

Insert this script into template
  <script type="text/javascript" src="http://www.google.com/recaptcha/api/js/recaptcha_ajax.js"></script>

The below one just to create the reCaptcha object
  Recaptcha.create("your_public_key",
    "element_id",
    {
      theme: "red",
      callback: Recaptcha.focus_response_field
    }
  );
Upon calling, the above code will create an image and 2 input forms, namely "recaptcha_response_field" and "recaptcha_challenge_field". The challenge field is hidden and contains a string to identify the challenge on the Google server.

- Create an ajax to send these 2 inputs to your server.

At the server side:

  
recaptcha_challenge=self.get_argument('recaptcha_challenge','')
  recaptcha_response=self.get_argument('recaptcha_response','')
  recaptcha_http=tornado.httpclient.AsyncHTTPClient()
  recaptcha_priv_key='xxxxxxyyyyyyyyzzzzz'
  remote_addr=self.request.headers['X-Real-Ip']
  recaptcha_req_data={
   'privatekey':recaptcha_priv_key,
   'remoteip':remote_addr,
   'challenge':recaptcha_challenge,
   'response':recaptcha_response
  }
  recaptcha_req_body=urllib.urlencode(recaptcha_req_data)
  
  try:
   recaptcha_http.fetch('http://www.google.com/recaptcha/api/verify', self.callback, method='POST',body=recaptcha_req_body) #https://developers.google.com/recaptcha/docs/verify
  except tornado.httpclient.HTTPError,e:
   print 'Error:',e
- Because the my Tornado is under Nginx proxy, so from the Nginx proxy I added a header (X-Real-IP) to forward the client's IP to Tornadoes.
- The above code will send to Google's API and Google will return a response with 2 lines:

true
xxyyyyzzz

- In the self.callback, you can first split the response by "\n", to obtain true or false value.
- Then you can perform your subsequent operations.

Wednesday, January 2, 2013

Using upload progress module in Nginx with Tornado

Previously I have set up a working upload conf.
I am following the official tutorial, http://wiki.nginx.org/HttpUploadProgressModule

1) Insert these into proper position in your Nginx.conf


upload_progress proxied 1m; #place under http{ ...}


track_uploads proxied 30s; #In my example, my upload button on the html template points to /upload (i.e. location /upload {...} , so I insert it in location /upload {...}


        location ^~ /progress {
            # report uploads tracked in the 'proxied' zone
            report_uploads proxied;
        } #Put this as a new location block


       #For see the effect of upload progress bar, I suggest u temporarily limit the upload rate by add upload_limit_rate 20000 in location /upload {...}



2) In your html template, just copy form the official doc


<form id="upload" enctype="multipart/form-data"
action="/upload.php" method="post" onsubmit="openProgressBar(); return true;">
  <input type="hidden" name="MAX_FILE_SIZE" value="30000000"  />
  <input name="userfile" type="file" label="fileupload" />
  <input type="submit" value="Send File" />
</form>

-- Change ...action="/upload.php"... to your destination


<div>
 <div id="progress" style="width: 400px; border: 1px solid black">
  <div id="progressbar" style="width: 1px; background-color: black; border: 1px solid white"> </div>
 </div>
 <div id="tp">(progress)</div>
</div>

-- Just copy it


 interval = null;

function openProgressBar() {
 /* generate random progress-id */
 uuid = "";
 for (i = 0; i < 32; i++) {
  uuid += Math.floor(Math.random() * 16).toString(16);
 }
 /* patch the form-action tag to include the progress-id */
 document.getElementById("upload").action="/upload.php?X-Progress-ID=" + uuid;

 /* call the progress-updater every 1000ms */
 interval = window.setInterval(
   function () {
     fetch(uuid);
   },
   1000
 );
}

function fetch(uuid) {
 req = new XMLHttpRequest();
 req.open("GET", "/progress", 1);
 req.setRequestHeader("X-Progress-ID", uuid);
 req.onreadystatechange = function () {
  if (req.readyState == 4) {
   if (req.status == 200) {
    /* poor-man JSON parser */
    var upload = eval(req.responseText);

    document.getElementById('tp').innerHTML = upload.state;

    /* change the width if the inner progress-bar */
    if (upload.state == 'done' || upload.state == 'uploading') {
     bar = document.getElementById('progressbar');
     w = 400 * upload.received / upload.size;
     bar.style.width = w + 'px';
    }
    /* we are done, stop the interval */
    if (upload.state == 'done') {
     window.clearTimeout(interval);
    }
   }
  }
 }
 req.send(null);
}

-- Copy the above too and you are done.


Saturday, December 22, 2012

Communication between Nginx upload module and Tornado, Python

Basic communication flow:

User (Internet, Port 80)===upload===>Nginx (listens 80) saves files===redirect modified POST headers via port 8080===> Python Tornado  (listens to 8080)====> return users information

1. Setting up Nginx

- I put the upload form (index.html) on /var/www/html/php, so I set this as the working directory (bad naming, but nevermind!)

==========
Index.html (from the Official example)


<html>
<head>
<title>Test upload</title>
</head>
<body>
<h2>Select files to upload</h2>
<form name="upload" method="POST" enctype="multipart/form-data" action="/upload">
<input type="file" name="file1"><br>
<input type="file" name="file2"><br>
<input type="file" name="file3"><br>
<input type="file" name="file4"><br>
<input type="file" name="file5"><br>
<input type="file" name="file6"><br>
<input type="submit" name="submit" value="Upload">
<input type="hidden" name="test" value="value">
</form>
</body>
</html>



Nginx.conf:



worker_processes  20;

error_log  /var/log/nginx/error.log debug;

working_directory /var/www/html/php;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    server {
        listen       80;
        client_max_body_size 100m;

        # Upload form should be submitted to this location
        location / {
        root /var/www/html/php;
        index index.html index.htm;
        }
        location /upload {
            # Pass altered request body to this location
            upload_pass   @test;

            # Store files to this directory
            # The directory is hashed, subdirectories 0 1 2 3 4 5 6 7 8 9 should exist << You must manually create these subdir before you upload, because Nginx upload does not support creating new directories

            upload_store /tmp 1;
         
            # Allow uploaded files to be read only by user
            upload_store_access user:r;

            # Set specified fields in request body
            upload_set_form_field "${upload_field_name}_name" $upload_file_name;
            upload_set_form_field "${upload_field_name}_content_type" $upload_content_type;
            upload_set_form_field "${upload_field_name}_path" $upload_tmp_path;

            # Inform backend about hash and size of a file
            upload_aggregate_form_field "${upload_field_name}_md5" $upload_file_md5;
            upload_aggregate_form_field "${upload_field_name}_size" $upload_file_size;

            upload_pass_form_field "^submit$|^description$";
        }

        # Pass altered request body to a backend of 8080
        location @test {
            proxy_pass   http://localhost:8080;
        }
    }
}


Tornado (rx.py)


import tornado.httpserver, tornado.web, tornado.ioloop, os.path

class requestHandler(tornado.web.RequestHandler):
print "Python receiver running"
def get (self):
print "Someone accessed through Get"

def post(self):
print self.request.arguments

#defines application of Tornado
#Pass any requests (post/get) to this handler 
application=tornado.web.Application([
    (r'/upload',requestHandler),

],)


#start the tornado, listens to 8080
if __name__=='__main__':
    application.listen(8080)
    tornado.ioloop.IOLoop.instance().start()


====================
1) Type: python rx.py
2) Goto http://localhost:8080/upload, get back to check if terminal shows "Someone accessed through Get" -->to show python working properly
3) Type: sudo nginx
4) Goto http://localhost/   --->should show an upload form
5) select file, upload ---->should show a blank page
6) go back to terminal, python interpreter should show something like this:

{u'file1_content_type': ['image/jpeg'], u'file1_md5': ['7fb958cd71e98537dd8e751d16f04ab9'], u'submit': ['Upload'], u'file1_name': ['1 copy.jpg'], u'file1_path': ['/tmp/9/0040448919'], u'file1_size': ['209926']}

1 copy.jpg is my uploaded file name. The key "file1_name" is the name of your upload form + suffix (_name) added by Nginx.

PS. Oh the file permission stuff may cause troubles. Pls change to 777 by chmod for testing purposes (remember to set to a safe value before into production!!!)

==End==

Friday, December 21, 2012

Installing NginX on Linux through yum

The OS I am using: CentOS 6

$ sudo yum install nginx

After that, nginx will be installed at this path:

/etc/nginx

To start, simply type: 

$ sudo nginx

If errors: bind to 0.0.0.0:80 failed (98: Address already in use), most likely due to the occupation of socket by apache. Stop apache by:

$ sudo service httpd stop

and retry.


Saturday, December 15, 2012

Hosting Ragnorok Online Private Server using rAthena

OS: Linux (CentOS 6)
Installation date: 14/12/2012
SVN: 17020 ==>This SVN uses 2012-04-10aRagexeRE.exe (we will talk about it later)
Version of RO episode: 14.1

Server Side:

Please follow the tutorial here:
*Please note that the last line of importing sql

mysql -u root -p rathena_log < logs.sql

Should be 

mysql --user=root -p rathena_log < logs.sql

Then everything should go fine.
Type 

cd ~/trunk
./athena-start start

to start your server.

=============
Client Side:

It took me a while to find the original RO game programme.
You can download RO Ep 14.1 here:
or 

Install it.

On your testing client machine (Windows), install SVN by going
download and install.

To download a patcher for generating the main RO exe file, use this ShinsDiffPatcher
open your command line in Windows, type

svn checkout https://subversion.assembla.com/svn/weetools/trunk/ShinsDiffPatcher/ <Replace here your path to download this>

You have to download a Renewal file, and patch it by ShinsDiffPatcher to generate main RO exe.

I tried this

But later I encountered errors using this. There are many people online saying having errors on server as below, and cannot login:

clif_parse: Disconnecting session #7 with unknown packet version (p:0x464d,l:19).

So, I tried other versions. This works. I suggest from the beginning we use this.

Open ShinsDiffPatcher, Choose WeeDiffGenerator from bottom menu, 
"Source executable": "2012-04-10aRagexeRE_J.exe
Choose automatically select. Untick "Read data folder first"
Then "Patch it"-->generated a exe file.

Next we are gonna generate a data file for our "in-house made exe" to use. 
Use GRF builder

Download 
(Version 3.0)

Unzip, 
Edit data/clientinfo.xml, modify the address to your IP

<address>123.123.123.123</address>

Then follow the tutorial from this video on using grf builder (use "data" folder to generate, not including folders like "AI" etc)
This generates a yourfilename.grf

In your official RO installation folder, find data.ini, add 0=yourfilename.grf so that it looks

[Data]
0=yourfilename.grf
2=data.grf

save.

Put the patched EXE and GRF file together to your RO folder. 
Start your server.
Start your EXE file. Bingo.


===========Materials with unknown use============
http://svn6.assembla.com/svn/ClientSide/Packets/Packet_db/

=====Debug======

Q1 # Can play in the LAN, not through the Internet
Reason 1) Did not forward the port if your server is inside the router
---> Forward the 3 ports 6121, 5121, 6900 to your server computer

Reason 2) Edit the Conf files
---> char_athena.conf


// Character Server IP
// The IP address which clients will use to connect.
// Set this to what your server's public IP address is.
char_ip: xx.xx.xx.xx <<your WAN IP

---> map_athena.conf


// Map Server IP
// The IP address which clients will use to connect.
// Set this to what your server's public IP address is.
map_ip: xx.xx.xx.xx<<your WAN IP

Reference to
http://rathena.org/board/topic/58572-probem-on-connection-to-server-with-wan-ip/