Blender Game Engine FPS Arms Movement and Basic Shooting Rig

This is a quick stand-in post that will be fleshed out later, hopefully :D

I am creating a video tutorial series for the Blender Game Engine, specifically for an FPS Arms game (where you see your arms and the gun, not just the gun) since I did not find many helpful tutorials on this.

The entire playlist can be found here:

http://www.youtube.com/playlist?list=PLb-cBpEOXTh8kPji_yaVGy6UyCncRjyid

The blenderartists link can be found here:
http://blenderartists.org/forum/showthread.php?278744-BGE-FPS-Arms-Video-Tutorial-Series-and-Example-Blend&p=2284833#post2284833

The download is:
https://docs.google.com/file/d/0Bz170qNApm2AZGc2RnBOZnYxdjQ/edit


What this currently contains:

An FPS rig (grouped) that has the following features:

A mouselook script by riyuzakisan
A rigged set of FPS arms (I removed and edited from a makehuman model)
A heavily edited and rigged SPAS-12 (from veti)
Mostly Python based movement and animation system (loosely based on the social FPS template)
Very basic running, walking, firing, idle animations
Bullethole and Bulletforce script (from the social FPS template)
The very very start of a menu system (very WIP)

The controls are as follows:
WASD for movement
SHIFT for run (hold down)
LCTRL for crouch (toggle)
RMOUSE for iron sight
LMOUSE for fire
SPACE for jump



My original sources/stuff:
MakeHuman:
http://makehuman.org
Social FPS Template:
http://www.mediafire.com/?cz5a1mv7c5j5782

Python mouselook script:
http://riyuzakisan.weebly.com/mouselook-script.html
Spas-12:
http://www.blendswap.com/blends/weapons/spas-12-lowpoly-gameart/

Blender to UDK FBX Skeletal Mesh 1p Arms and Weapon WorkFlow

Posting my notes here so that people can learn from my mistakes...

Was hoping for some input on improving my blender to UDK 1p (1st person) arms + weapon animations. My workflow is as follows:

Blender Setup

Got some good information here: http://krisredbeard.wordpress.com/tu...-player-model/

Keep these suggestions in mind.

However, for whatever reason, the scale settings he sets up do NOT work for me at all, still WAY too small in UDK viewer, but I do edit the fbx exporter script for blender as he suggest.

Blender Units

I do set up my scale for blender (using blender units - the default) with Grid Floor Settings as follows:
Lines: 100
Scale: 8
Subdivisions: 8

Using these settings, my arms are ~95 blender units (BU) long from fingertip to fingertip. ~8 BU along the Y axis, and ~8 BU along the Z. (The shoulders are almost round on my model)

The shotgun I'm using is ~52 BU long.

Mesh and Armature Requirements

Once I have the scale set, I use CTRL+A to "bake" the scale in blender. Otherwise your size won't change at all in UDK. In blender, under the "N" toolbar, the scale MUST read 1.00 for everything.

My armature has a root bone at 0,0,0. I don't think it has to be called root_bone, but that is what I name it. I then have my basic armature setup with IK constraints. I also have 4 IK bones, used to control the hands and elbows. I have my two Hand IK bones as CHILDREN of my root_bone. (Right now I wait to parent them until just before export, seems to work fine). I want to use them as Sockets in UDK.
None of my IK bones have "deform" checked in the bone tab in blender.

DO NOT do anything fancy (ie blender specific) stuff with your armature. IK constraints sure, custom shapes for IK bones, sure, but nothing like curved/segmented bones. Those won't work when exporting. Trust me, I almost cried about not being able to use curved bones when animating.... much more difficult. If anyone discovers how this could work in UDK you would be my hero.

Setup a Reusable Scale Object 

Once that is all set up, I usually export it several million times to get the scale exactly right in udk. Then I usually append (SHIFT+F1) a 3p mesh, size it so it looks correct, then create a box that shows the basic size of a human in UDK. That way I have a "measuring stick" I can pull up at any time. Using "M" I move those measuring sticks to another layer, or save them to a separate blend file. This way I can easily append these objects to any blend file to check the sizing of my models.

Shotgun Armature Settings and Difficulties

My shotgun armature has 6 bones, but should contain a:
root_bone - controls the entire gun
muzzle_flash - used in UDK for creating a muzzle flash
along with your usual stuff like trigger, slide, etc.


This is where I have issues.
I can't get the shotgun to appear correctly socketed in UDK (I have to manually set it up in UDK)
I have tried setting root bone to 0,0,0 then rotating/moving the gun so it would look correct in the human hand.
I have tried exporting it with the root bone offset of 0,0,0 (by 30 units or so along the X axis)
Nothing seems to work, but at least I can adjust the offset in UDK
If someone knows how to fix this, that would be great.

I usually export the gun as a skeletal mesh now (don't export animations)
In object mode, select the GUN MESH AND ARMATURE, both must be selected.
Again, make sure your scale is baked.
For my fbx export settings:

Check "Selected Objects"
Leave scale at 1.00 (It didn't even seem to ever work for me?)
Leave forward and up at default
Just select Armature and Mesh, nothing else
"Apply Modifiers" should be checked
Only Deform Bones I usually leave unchecked, since my IK bones aren't deformed.
Uncheck Include animation.
The rest should be self-explanatory.

Animating the Mesh

Then I start Animation, which unfortunately I am weak at in Blender, so if someone could improve my knowledge of this that would be great.

Then I take the root_bone of my shotgun, and use an IK constraint to child the root_bone to my right_hand IK. Then I clear inverse, and clear the "scale" settings for the IK constraint and in object mode position the gun so it looks correct in the hand. This means when I move or rotate my hand IK the gun moves with it.

I first create a "Base_Pose" Action that is just my mesh in the default arms out pose.

Next I create a new action, do my animation, save my file.
For each UDK animation I want, I create a new action. Udk will import each action is as a different animation.

I think go back to the Base_Pose action (arms out) and select my arms armature and mesh, then choose export.

First I just export it as a skeletal mesh, and you can either uncheck include anims on fbx export, or import into udk. I usually do it on export, for a smaller file size, keeps things nice and neat.

I then import it into UDK (check Use TOAs Ref Pose).

I again double-check the size in UDK, make sure it looks correct. If you parented the Hand IK bones to the root bone, they should show up in your skeletal mesh. All the other IK bones you did not parent should not show up. I create a socket in the R Hand IK, and add my shotgun as the skeletal mesh. I then spend an annoying few minutes setting up the offset so it is in the hand correctly.

Animation Exportation Gotchas

Then go back to blender and for some reason (I may have done something wrong in blender?) you can't just export again, with animations.

I have to switch back to animation mode in blender, select my animation I created, and scrub the slider to make sure it moves (sometimes switch to pose mode and back). Then I can export the animations. If I don't do this, when I import the animations, they are only one frame (which is odd, since they show up fine in the Autodesk converter).

Speaking of which, to update your blender fbx files to the newest version, run them through the free Autodesk converter.

I then import the animations into a new animation set (uncheck the anim rotation only setting on import) and test. They should work fine.

The big gotcha

Currently, for some unexplained reason, UDK will import all of your blender actions at one single frame range... whatever you had set in blender, so all of your animations will be the same length.  Which doesn't work. Some people said it works fine in the December 11 and May 12 UDK downloads, but it certainly doesn't work with the June and Nov 12 UDK downloads, as I have tested both.

So you need to use Autodesk FBX Viewer to open your fbx files, use the take manager to select the takes you want to keep, then chose export takes as individual fbx files.  Then when you import them into UDK they will have the correct number of frames.

That is the basic workflow I use in blender for fbx animation.

I am hoping everyone can either learn from this or give me suggestions on how to improve this, as I have almost no experience with either UDK or blender, just started a few weeks ago.

Using Phpseclib Read/Write to Manage the Terminal

About Phpseclib

Managing terminal commands with php can be a tricky business.  You have several built-in php commands, such as exec(), system() and passthru().  The issue with these commands is that you will be running all of these commands as the www-data user (or equivalent).  No sudo commands can be run, and you cannot log on as another user.  As well, commands are "one-offs", no interaction with the shell is possible.

You could install the PHP SSH2 library, and run shell commands that way... however, if anyone else would like to use the software, they will have to add that php library as well, since it is not a default library.  The other option you have is phpseclib. Phpseclib is a pure php implementation of SSH2 (and tons others).

In this tutorial we will be covering the SSH2 features of phpseclib, specifically read()/write(), since they can be slightly complicated.

Setting up phpseclib

Extremely simple to do:  download either the stable version (currently 0.3.0) or (as I recommend) pulling the latest version from git: https://github.com/phpseclib/phpseclib.git.  The latest version seems to work much better for me.

Place the folder in your php project folder.  Next, in your php page, include the following code.

set_include_path(get_include_path() . PATH_SEPARATOR . 'phpseclib');

include('Net/SSH2.php');

Next, you need to actually set up an ssh connection to your server in your php script.  Usually this would be localhost.  This is done like so:

$ssh = new Net_SSH2('localhost'); //starting the ssh connection to localhost
if (!$ssh->login($username, $password)) { //if you can't log on...
    exit('Login Failed');
}

If you don't receive an error message, congratulations, you are logged in, that simple.  The $username and $password are actual PHP variables that can be collected from user input (Be careful!  That is sensitive information...)

Phpseclib exec() vs read()/write()

Phpseclib has two different ways to execute shell commands.  The first is exec(), which is simple and effective, and looks like so:

$ssh->exec('killall -v apt-get');

Just make sure that whatever variable you named your connection (in this case $ssh) you use when you execute the command. If you want, you can also store the output of the command like so:

$output = $ssh->exec('killall -v apt-get');

Then you can echo or search the output, which is very useful. Essentially, the exec() command logs on, executes command, then exits. For example, the following command DOES NOT output the contents of "/etc/test":

$ssh->exec('cd /etc/test');
$ssh->exec('ls');

This is because the second command does not follow the first. The connection at the end of the exec command is closed, then reopened for the next command. The ls command will list the contents of the default logon folder (Usually the home folder).


However, the read command could accomplish this... and much more.

Using read() and write() together

IMPORTANT NOTE:  The write() command must simulate the enter key, so end all of your command with "\n".

The read() and write() commands must be used together... the best way to show you is with an example....

$ssh->read('/.*@.*[$|#]/', NET_SSH2_READ_REGEX); //start by reading for the command prompt using regex.. we COULD use our username variable in here to make it even better..
$ssh->write("sudo sed -i 's/KEY_PROVINCE=.*/KEY_PROVINCE=\"$key_province\"/g' $var_file\n");
$ssh->setTimeout(10); //right before the read set timeout so php don't crash/timeout on unexpected output
$output = $ssh->read('/.*@.*[$|#]|.*[P|p]assword.*/', NET_SSH2_READ_REGEX); //reading for either the password or the command prompt
echo "$output
";
if (preg_match('/.*[P|p]assword.*/', $output)) { //if we read a prompt asking for the sudo password
 $ssh->write($password."\n"); //write our password (Stored in $_SESSION) to the prompt and "hit" enter (\n)
 $sed_output = $ssh->read('/.*@.*[$|#]/', NET_SSH2_READ_REGEX); //make sure our password worked, read for the command prompt.....
 echo "
Entering SUDO Password... Check for errors....: $sed_output
"; //echo the output of the command... this usually will print errors/output of command...
}
$ssh->read('/.*@.*[$|#]/', NET_SSH2_READ_REGEX);//Both reads' appear to be required! 
$ssh->read('/.*@.*[$|#]/', NET_SSH2_READ_REGEX);//Both reads' appear to be required!
echo str_repeat(' ',1024*64);//purging output to the browser

Lets go through the steps involved:
Line 1.  The first $ssh->read is using the REGEX flag, which means it accepts regex commands.  What we are doing here is "reading" what the SSH terminal says currently.  Since we have just logged on, it should be at a command prompt.  So I am using a regex that says "Any number of characters, followed by an "@" symbol, followed by any number of characters, follow by either a "$ or #".  For those of you familiar with the command prompt, you will realize that the regex should work for recognizing almost all command prompts. Once we have read the command prompt, we know it is ready to accept commands.
Line 2.  Now we will write our command, a simple sudo sed command.  You can put php variables in the command as needed.
Line 3.  Set the timeout, so we don't have to wait for php to time out if something goes wrong.
Line 4.  Now we are going to "read" the output from the command, so we save the next ssh->read command to "$output".
Line 5. Just echoing the result of the last write command (this has to be done after the "read" because phpseclib needs to read the result of the last command)
Line 7.  Now we are going to search/read the resulting output to see what we find.  We will once again use a regex, except this time we will test for two outputs.  A) The regular command promt OR B) Password.  As you know, since this is a sudo command, we might get prompted for our sudo password.  If that happens, we need to handle that.
Line 8.  We will reach this command IF either A) or B) happened.  Then we will do a preg_match, to see if the output contains the word "Password". If so, we will write the password to the command line, then read for the command prompt again.

We finally reach the end of the password loop, and next we do a final $ssh->read to get the final results.

IMPORTANT NOTE:  Before launching directly into another $ssh->write command, make sure you add ONE MORE $ssh->read command before doing so, both appear to be required.

Phpseclib Logging

You can view the complete phpseclib log by adding the following lines:

define('NET_SSH2_LOGGING', NET_SSH2_LOG_COMPLEX); //add near include lines

echo $ssh->getLog();//add after the command you want to see the log for....

This will output the log to your browser.

Good luck!

Theories on how Sherlock Survived the Fall

FYI, this contains spoilers for those who have not seen Sherlock Season 2 Episode 3.

In Sherlock Season 2 Episode 03 "The Reichenbach Fall" Sherlock appears to fall to his death off of what appears to be a 4-5 story building.  Later in the episode, Sherlock is seen, alive and well.  The question is, how did Sherlock survive the fall?

Before I delve into this, I will present this caveat.  Film and TV makers are free to make decisions in their story lines that do not hold with reality, or at least stretch it to the outer limits.  The viewer is forced to make a logical assumption based on the facts presented in the show.  To date, however, Sherlock has done a reasonable job of sticking within those limits.  I was extremely happy when Moriarty's computer code was not real... that would have been a distressing separation from reality, and it shows that the writers are doing their best to stick to reality.

Now, on to the fall.

There are several concrete clues as to what happened, and several possible hints, although we are forced to speculate on what they mean.

The Final Problem

Let's start at the beginning.  Throughout the show, Moriarty constantly refers to "The Final Problem".  "What's the final problem?  I did tell you, but did you listen?" Moriarty tells Sherlock after he walks out of jail. "I want to solve the problem.  Our problem.  The final problem."  Cursory speculation would lead to the conclusion that Moriarty was talking about getting rid of Sherlock.  Especially since Moriarty mentions "The Fall" directly after the problem.  I believe that most people, myself included thought that Moriarty was mentioning a metaphorical fall, a fall from grace.

The Fairy Tale

Throughout the show, there are numerous references to Grimm Fairy Tales, most notable Hansel and Gretel, which was the motif in the case that Sherlock solves (almost an afterthought in this episode!).  As well, Sherlock receives a burnt gingerbread man, which he appears to interpret as Moriarty wanting to "burn him".

However, there is another very interesting reference. When on the rooftop, Moriarty says to Sherlock "I love newspapers.  They're like fairy tales, and pretty grim ones too."  Why did Moriarty feel the need to make this reference?  Another throwback to the gingerbread man?  Does Moriarty fancy himself as the fox that caught and killed Sherlock? Or another fairy tale entirely?

Sherlock's Ignorance?

Once on the rooftop Sherlock appears confused when Moriarty says "Glad you chose a tall building, nice way to do it."  This appears to be an act, to confuse Moriarty, as Sherlock told Molly that he was afraid that he was going to die.  He surely knew that Moriarty wanted him to die, and was acting to cover it up to Moriarty.  I am however, confused as to why Moriarty bought this act.  Sherlock had chosen the location of the building, surely he must suspect that Sherlock had a plan, especially when Sherlock met him on a rooftop.  He should have surmised that Sherlock had realized he had meant a literal fall, not only a fall from grace, and would have had time to create a plan.  Perhaps it was merely arrogance?  He could not imagine a way out, so Sherlock could not imagine a way out?

In fact, I think Sherlock is many steps ahead of this from early on.  He forces John Watson to take another taxi when leaving the police station... an odd choice.  The reason Sherlock gives is that John Watson might talk and disturb him, but the taxi contains Moriarty... did Sherlock know this in advance?  It doesn't appear so, but an interesting tidbit.  As well, when Mrs. Hudson is "shot", Sherlock immediately acts unconcerned, and refuses to go see her. I believe Sherlock immediately knows this isn't true, most likely because he set this up to separate John from himself long enough to get things done.

Molly is the key here.  Moriarty had everyone one of Sherlock's friends watched.  Except Molly, which is ironic, since he "dated" Molly to meet Sherlock for the first time.  Sherlock had always treated Molly poorly, which allowed him to use her to complete his faked death.

Moriarty's Role

As for Moriarty, I firmly believe he is dead, and will stay that way.  There are several reasons for this.  One: Moriarty did not pull any punches for this entire setup.  He exposed himself to the media, the world, and then claimed to every single rouge nation and intelligence agency that he had a key to unlock anything.  As smart as he might be, I think Moriarty knew his life span would be limited once they realized he was lying.  Two:  Moriarty was, simply, a psychopath.  When Sherlock speculated about what Moriarty wanted, I think the answer was simple: a nemesis.  He wanted to know that Sherlock was not ordinary, was a worthy opponent. He wanted to go out on the top.

When Sherlock was talking to Moriarty on the roof, the final confrontation before Moriarty shot himself, Moriarty accuses Sherlock of being ordinary. Sherlock replies that "You want me to shake hands with you in hell, I shall not disappoint you."  Moriarty once again accuses Sherlock of being on the side of angels, but after staring at Sherlock for some time, says "No, you're not. I see. You're not ordinary, no, you're me. You're me. Thank you! Sherlock Holmes.  Thank you.  Bless you."  What changed his mind?  After viewing the scene multiple times, I cannot determine what changed his mind.  He is looking at Sherlock, he cannot have seen something in the background that betrayed Sherlock had a plan for surviving.  What did Moriarty mean by "You're me"?  Did that mean Sherlock was as smart as him?  That he realized Sherlock had outsmarted him?  Did he mean Sherlock could be as unscrupulous as him?  If he had realized that Sherlock had outsmarted him, why kill himself?  I don't think that fits his MO.  Or was Moriarty truly afraid that Sherlock would guess the "kill code" that would call off the snipers?  Did Moriarty look into Sherlock's eyes and fear that Sherlock could outguess him and decide the only way to make certain was to shoot himself?  I doubt the last one... I don't think the shooting was spur of the moment at all.  It was very planned, and I think that Moriarty had planned on both of them dying there on the roof, or the sidewalk below.

People have been saying that if Moriarty had truly been killed his head would have been blown up from the bullet, which is not true.  A small handgun like a glock or 9 mill would not leave a massive exit wound, and would not necessarily have bled excessively.

The Jump

Now, onto the scene.  Here is a rough drawing of the scene where Sherlock fakes his death.  To see it in google maps, see here.

A few notes.  First, as far as I can tell, the sniper is directly behind Watson in the building across the street.  Due to the timing, he must have been there before Watson arrived so he might have been the sniper that was ordered to watch Sherlock fall, or he may have been the sniper on Watson.

If you take a look at the picture below, you can see an open window which might contain the sniper (although there are white blinds in the window, which does not show up later, when we see the sniper.)

Either way, it appears the vantage point of the sniper is approx. the same vantage point as John Watson's, although slightly elevated.  This means the small building (and trash truck) could have been obscuring his view of Sherlock's body as well as Watson's view.

Now on to why most of the current theories are wrong.  (Logically speaking.. as I mentioned TV shows tend to break this).

The Theories

Sherlock threw Moriarty's dead body off the roof/or a dummy

When Sherlock meets the little girl that was kidnapped she appears to recognize him, prompting Sherlock to think there was either someone wearing a Sherlock mask, or a look-alike.  The theory is that Sherlock either found a mask on the body, or just threw the body off the roof without a mask.  John was fooled because the biker knocked him over and drugged him with the hallucinogenic drug from "The Hound of the Baskervilles".  And Molly fakes the autopsy.  

Here is why that is wrong:  The camera would have to lie to us.  We see a body flinging their arms around as it falls.  Not a dead man.  I don't think the writers would do that to us, and I would hate them.  Too much of that is such a stretch, and requires us to not believe our eyes.  The dummy hypothesis is bad because John did feel the wrist, he should have been able to tell the difference between a dummy and a human being, even confused as he was. 

As well, every single onlooker would have to be a paid actor... although at least some may have been, since they appear to hold John back from seeing Sherlock.

Sherlock threw Moriarty's body down, then jumped down into the back of a truck and swapped bodies when John was knocked down.

As you can see in the photo, there is a garbage truck of some kind waiting at the curb, and it pulls out after the body is on the ground, which seems to be a very odd thing to do, unless the truck had to be there for Sherlock to jump.  However, I don't like this theory.  In that case, two bodies have to fall off the roof, one before Sherlock unless someone else throws the other body... which means a body would have been on the street before John shows up.  Not easy to do without drawing attention.

Sherlock survived the fall and had John knocked down to disorient him enough to miss a pulse.

This one is interesting, it just posits that Sherlock used wind resistance and some sort of airbag to break his fall enough to survive and then added blood while John was knocked down. This seems unlikely, as one would usually have some sort of massive damage, broken bones, punctured lung from a fall, and may not be well enough to add the blood.  It is the easiest explanation as it (if done right) would fool some of the onlookers (not the close ones), John, and the sniper very handily, although the paramedics would have to be silenced.  I don't like this one either... too risky that Sherlock would die from the attempt.

Sherlock jumps into the back of the truck, then gets out and onto the sidewalk before John sees him.

Not a huge fan of this one.  There is a time lapse of approx. 8 seconds when Sherlock's body appears to hit the ground and when John Watson rounds the corner and sees the body THEN is hit by the biker.  Seems hard to do in that time lapse, but I suppose it is possible.  (In the screen below, notice the truck is now gone.)

Its one massive conspiracy

Moriarty tells Sherlock, "I knew you'd fall for it.  That's your weakness, you always want everything to be clever." 

 So I must admit that this isn't a clever solution by any means, but perhaps Moriarty said that for a reason... to make us realize that the cleverest solution isn't always the best one.

This obviously sounds the best, as a massive conspiracy is required to make this seem believable.  Everyone on the street is an actor (or a homeless person), when Sherlock jumps, he lands in the back of the garbage truck (hard to do, appears to be a 10 foot distance from the building to the truck).  A body is placed on the ground at the same time, or thrown from a second story window where John cannot see.  If you take a look at the picture above (with the garbage truck) you can see from John's point of view that a body is on the sidewalk, and this is before John is knocked down by the biker. However, the body is far enough away that John cannot tell who it is.  Once John is knocked down, Sherlock gets out of the truck and swaps places with the body, and the truck carries the fake body away.  John is disoriented and cannot read the pulse or Sherlock is using a variety of tricks to slow his pulse down significantly (Sherlock is seen with a rubber ball).  Molly fakes the autopsy, and bang, alive Sherlock.


Either way, I have enjoyed Sherlock immensely, and cannot wait for 2013.

PHP Real-Time Search Using MySQL and AJAX

Recently I decided that instead of posting all of my links directly on my homepage (It was becoming a long list, and annoying to have to edit the HTML every time to add a link) I would create a php based search engine, that used AJAX for real-time results.

After using Google for an extended period of time, finally found my first link: http://woork.blogspot.com/2007/11/simple-search-engine-in-ajax-and-php.html  (I DON'T recommend this unless you know what you are doing, it is full of errors/missing syntax/db connections).  After following this guide (and fixing mistakes) I realized this did not work as I had hoped.  When I searched for my keyword, it did not appear to matter what keyword I searched it pulled all the results up.  Since I am not a php expert, went back to Google and found the following article: http://blogs.nightglass.com/jeremy/2008/07/23/realtime-search-with-php-mysql-and-prototype/

The second article was a much better guide on how to set up real time search, now I just had to edit it to my needs... here is what I wanted.

A search engine that would search my MySQL database of internal website links based on 3 keywords that I specify upon creation of the link.  So I created a MySQL database, lets take a look at the layout.  (Please note I am not even close to proficient with MySQL, which is why I'm using phpmysql).

MySQL Database Setup

The first field is the Index, a simple auto-increment field just to make my links have a unique number.  The next field is "link_text" which is just the html link.  Then I have my three tag fields, the next is the current timestamp, if for example I wanted to show the latest links that were added.

Form.html - The search box setup

The "Form.html" sets up the basic search box, and calls the javascript file, lets take a look:

 
  
  
 
 
  Keyword Search:
  


  



  
 

Fairly simple, ignore the first half, that just is styling for the search box.  The part we need starts at "from id="searchform"".  That creates the search box.  Notice "onkeyup" this updates the results every time a key is let up.  The "name=searchq" is also tied to our next php script, so remember that name.

Prototype.js is the javascript that creates the search results, which can be found here.

Search.php - Where the magic happens

Lets take a look at our "search.php" document, which is the main part we need to know.

 


"; //this div is used to contain the results. Mostly used for styling.
   
 //This query searches the name field for whatever the input is.
 $sql = "SELECT link_text FROM Links WHERE link_tag_1 LIKE '%$searchq%' OR link_tag_2 LIKE '%$searchq%' OR link_tag_3 LIKE '%$searchq%' ";
   
 $result = mysql_query($sql);
 while($row = mysql_fetch_assoc($result)) {
  $id = $row['link_text'];
  echo "$id";
                echo "br"; 
  }   
  echo "div";
 }
?>

*Note: I have not yet cleaned the input, so make sure you do that to avoid MySQL injection. Also, because blogger hates me, had to remove the formatting from the last div tag, and there is some odd formatting at the very top.

The first lines are standard, you are connecting to your MySQL server, connecting to your link database and table.

The interesting part is when you grab your search query from form.html "$searchq = $_POST['searchq'];". If it is empty, you will display nothing. Else, you will start getting search results. Next is your sql command. You are selecting the link_text that matches any of your tags for that link. The "%" signs specify any characters before or after the search term, so you don't have to type out the entire tag to get a match, you can type one letter and it will match any of the letters in all of the search terms, and will narrow it down as you type.

Next, we set $id to equal row "link_text" which is just the fully formatted html link, so this displays in your popup box. If you don't like the popup box, go ahead and delete or reformat it in Form.html, I did not like the look so changed it to a simple blue link that appears below search results.

EDIT: The files how now been uploaded to google docs, link to download here. Let me know if you have any problems.

Okay, you should be ready to go, happy searching. Next time I hope to post about adding/changing entries in your mysql database from a web page.

Backuppc Basic Setup and Configuration on Debian Lenny for Windows and Linux Hosts

Backuppc is an extremely robust and flexible backup solution for your entire network.  Unfortunately, the very complex and flexible nature of Backuppc makes it a very complicated solution to setup and get working correctly.  However, as usual, installing the application on debian is very easy:

sudo aptitude install backuppc

Unfortunately that is only the first step... once it is done installing, it will have you answer some setup questions about setting up the web interface.  It will also generate a password for the web interface that you will need to use to log on.
Once the setup is complete, you can visit the backuppc web interface through http://localhost/backuppc.  One of the errors you may be greeted with is "Error: Unable to connect to BackupPC server".  If that happens, it may be because backuppc is not running, or is attempting to run under the wrong user.  To fix this (in almost every case), you need to change your use to backuppc, and start backuppc again.

su backuppc
/usr/share/backuppc/bin/BackupPC -d

This should start backuppc as a daemon, solving your problem, and allowing you to once again revisit the web interface.
The web interface is laid out (in my opinion) in a slightly confusing way.

The view you will see when you first log on is an overall server status page.  Once you have hosts with backups, there will be some very useful graphs displaying pool storage size.  Good, you have the basic program up and running... now for a little bit of customization.

Changing the Backuppc Pool Directory

By default, all your backups are stored at /var/lib/backuppc.  However, I wish to store my backups on another disk drive that has a lot more storage than my system disk.  To do this we need to create a soft link from the default backup directory to whatever directory you wish to store the backups.  You will notice that we create two directories called "pc" and "cpool". "pc" is where all the files are stored, and those two directories were originally causing permission issues.
Changing the Pool directory:

cd /var/lib/backuppc
sudo rm -r *ln -s /var/lib/backuppc /new_drive/backuppc
sudo mkdir pc cpool
cd ..
sudo chown -R backuppc *

If you have problems, it may be a permissions issue:

sudo chmod -R 777 backuppc

I would recommend not leaving the permissions wide open... just make sure it is a permission error and slowly restrict the permissions until everything works correctly.  Now the backups should redirect directly to your new drive.

Adding Hosts

Now we need to add some hosts for backup.  Select "Edit Config" from the left menu.  Click on the "Hosts" tab.  Here we can add hosts for backup.  I personally have DNS reservations for all of my hosts, so I can add by IP, however if your hosts will have different IP's I would highly recommend checking the DHCP option.  As long as your hosts are registered in DNS backuppc can track them and back them up no matter where they are.  Add the hosts you will be backing up, hit save, and then click on the "Xfer" tab.

Setting Default Xfer Options (for Windows)

Next, we need to set the default transfer settings.  This is mostly a setting for large networks of pc's that are set up the same way, which makes it unnecessary for the admin to configure each pc individually. Click on "Edit Config" on the menu on the left, and select "Xfer" from the top menu.  How you set this up depends entirely on your network setup.  I have two Windows 7 PC's, so I will be setting the default settings for a SMB share.

The above screen shot shows the settings I set for the default.  I chose SMB for the transfer method.  The advantage of using SMB is that it is already built into Windows... which is good from an ease of use scenario, but is not as good from a security standpoint.  For the SMB share name, I added the administrative C$ share, which is the entire C$.  In this instance I also should have set default "Include/Exclude" settings.  This part is not extremely intuitive, and it took me a little while to figure that out.  Shown below is one of my host machines.

To ensure that only the right files are backed up, we need to add first the "SmbShareName" to the "New Key" field, and click "Add".  Once we have added the base share name, we can add the individual paths that we wish to backup.  You will notice above that I told backuppc to backup only the "Users" directory under the C$ share.  So now, instead of backing up the entire OS, backuppc will only backup "C:\Users", which is a much better option. You can obviously add more than one folder.  You can also add folders to exclude, which is a nice option as well. Notice that the "Override" box is checked... that just means that this host differs from the default configuration in this area.

Backing up the Localhost

Backing up the localhost is the easiest of all the pc's to backup (in my opinion).  The easiest way to backup your linux localhost is through "tar".  In the top host drop-drop menu, select "localhost".  Click on "Edit Config" which will enter the host-specific configuration.  Select "tar" and from the "XferMethod" drop-down menu.  Next, we need to select the tar share names that we wish to back up.  Since the localhost is not running a whole lot of services, and is not mission critical, I am only backing up the /var and /etc directories.  Go ahead and save the config and you should be ready to backup your localhost!

Backing up Linux Hosts with Rsyncd

I prefer the rsyncd method for backing up linux hosts.  It does involve installing rsyncd on the linux hosts in question, but it is a rather quick install.

sudo aptitude install rsync

Once that is done installing, we need to setup the configuration file.  On linux, the config file is called rsyncd.conf and is located at /etc/rsyncd.conf.. so

sudo nano /etc/rsyncd.conf

Now we copy the config in: uid = root pid file = /var/run/rsyncd.pid use chroot = no read only = yes transfer logging = true log format = %h %o %f %l %b log file = /var/log/rsyncd.log slp refresh = 900 secrets file = /etc/rsyncd.secrets [backup] path = / comment = backuppctest read only = no auth users = backuppc
Now we need one more file to complete the configuration. The file is called rsyncd.secrets, and is stored in whatever place you specified in rsyncd.conf. The file only contains one line

backuppc:Yourpassword

The format of the file is simple: username:Password. Make sure the username matches exactly what you specified in rsyncd under "auth users", otherwise you will have difficulties.
Next we need to secure the secrets file:

sudo chown root /etc/rsyncd.secrets
sudo chmod 600 /etc/rsyncd.secrets

Next, lets setup the backuppc side... shown below is a screenshot of the configuration.

The "RsyncShareName" is the share name that you specified in the rsyncd.conf file.  The username and password should match exactly what is in the rsyncd.secrets file, and make sure the "RsyncdAuthRequired" is checked.  You will notice in the config I posted above, I was very aggressive and made the share root be root.  That being the case, there are many folders I should exclude from backing up, since they are either temporary folders/files or files/folders I really don't want backed up (i.e. floppy).  As well, be very careful about hard/soft links as those can cause issues.
If you experience problems, there is one configuration issue that you can check on the backuppc server.
When using rsyncd for backuppc you will need the File::RsyncP perl module installed.  It may be installed already, but if not perldoc File::RsyncP, to make sure RsyncP is installed.
As well, the log file on the client machine is a great help as well, located at /var/log/rsyncd.log.

Restoring Files, Logging and Information

Well, you now have a local backup solution for all your PC's.  Now you can use backuppc's powerful logging and reporting functions to view the status of all your backups.  As well, you can easily browse your backups and restore files if needed.  To browse backups, just select your host, and in the left menu will be an option to  "Browse Backups".  Select this to browse and restore backups.

Congratulations on completing the basic Backuppc setup.  Hopefully in a later post more advanced topics can be covered. 

Udev "Error" in Debian Sid, Kernel Upgrade difficulties

So I have been hesitant for some time now to upgrade by Debian Lenny box, due to some upgrade errors I encountered in the past due to dependency conflicts.  (I have now switched to aptitude, which seems to do much better).  However, it was finally time.. I had several hundred updates to run, so it was time.  When I ran this installer, I received the following error:

Preparing to replace udev 0.125-7+lenny3 (using .../archives/udev_151-3_i386.deb) ... Since release 150, udev requires that support for the CONFIG_SYSFS_DEPRECATED feature is disabled in the running kernel. Please upgrade your kernel before or while upgrading udev. AT YOUR OWN RISK, you can force the installation of this version of udev WHICH DOES NOT WORK WITH YOUR RUNNING KERNEL AND WILL BREAK YOUR SYSTEM AT THE NEXT REBOOT by creating the /etc/udev/kernel-upgrade file. There is always a safer way to upgrade, do not try this unless you understand what you are doing! dpkg: error processing /var/cache/apt/archives/udev_151-3_i386.deb (--unpack) subprocess new pre-installation script returned error exit status 1

Manually Upgrade the Kernel

So, it was time to upgrade my kernel manually.. no way around it.  I downloaded the latest kernel for Sid (2.6.32-3-686 at the time of running) and manually installed, which I've never done (not too difficult, or so I thought)....

I then needed to reboot after the kernel installed, so that changes could be made.  I waited patiently on reboot for my server to come back on-line.... it never did.  I went over and hooked up a screen to my server, and noticed that GRUB had errored out.

Fix Grub

I noticed that the problem must be that for root it was looking at (hd0,1) and root=/dev/hdb1 (not possible).
So after changing (hd0,1) to (hd0,0) and root=/dev/hdb1 to root=/dev/hda1.  I'm not entirely sure why, however, if you do run into this issue, I would recommend that before you reboot, run this command:

 sudo update-grub

Generating grub.cfg ...
Found background image: moreblue-orbit-grub.png
Found linux image: /boot/vmlinuz-2.6.32-3-686
Found initrd image: /boot/initrd.img-2.6.32-3-686
Found linux image: /boot/vmlinuz-2.6.26-2-686
Found initrd image: /boot/initrd.img-2.6.26-2-686
done

This may fix your problem.  If you need to, you may manually edit your grub file to make sure it is correct.

 sudo nano /boot/grub/menu.lst

Haven't had a problem since!

Happy upgrading!