A couple months ago while I was trying to make some decent progress on Nuubz when I came to a point that I realized I needed to start and do some serious work on an unrelated project before I could advance what I wanted to work on. In particular, I wanted to move the system/site security forward, which I hope will lead to better comment spam prevention and better all around security. I already had a plan in place, I just needed to actually implement it.

Enter Project Indigo. Or at least *MY* Project Indigo.

Some years ago, while working at a web hosting company, I noticed that people kept trying to break into a site of mine for which I literally had no content. There was a single, simple HTML file saying “There’s nothing here yet.” So, I quickly wrote a script and database to record those attacks. That system has been tracking these attacks for 6 years. Last year, I noticed an abundance of brute force ssh attacks on the server as well, and started recording those in a separate system. I decided to put this data together in a security web site project to help the masses, and myself, but I just didn’t get around to doing it until Nuubz prodded me to do so.

So, I put an old domain name I owned to use and Project Indigo was born. I still have a lot to do on it, including actually providing some useful information beyond some statistics on the home page, but as you can see, it’s receiving live information currently from two virtual private servers. (I’m getting ready to shut one down, however.) There have been over 700,000 SSH attacks detected and reported to the system as of this moment, while only 2,700 “404” attacks. I emphasize “404” attacks because these are just pure page not found attacks; in my honeypot site, these are requests for pages that don’t currently and have never existed on the site, and don’t have any additional attack parameters. There’s another similar attack that I’m simply calling “web attacks” that aren’t yet reported, these are (again on my honeypot sites) page requests with GET, POST, and/or cookie values that were never requested, used, or expected on the site, regardless of whether the requested page has existed or not. (Again, on the honeypot site, most of the pages that have been targeted have never existed.)

I’m still debating whether I should try to make a business out of this or not, but I’m willing to accept donations. I’ll provide that information when I make it possible to register an account on the site and put a little more polish into it. In the mean time, some attack data is available on Google if you search for “site:prjindigo.com” and machine readable data on a given IP address is available at https://www.prjindigo.com/data/<ip address>.json . Both IPv4 and IPv6 addresses are supported though I’ve only seen a few v6 addresses enter the system at this point. (Be sure to URL encode IPv6 addresses.)

I have created a Github repo for the honeypot software, which is still in active development as well, and I’m working on a Go language program to report the data and possibly parse log files to get ssh failure data. (I’m still unsure about using Go to parse that data as the log files may change from OS to OS.) Don’t rush out and clone either repo yet, both depend on client identifiers and encryption keys that depend on having an account at the Project Indigo website, which, as I indicated above, isn’t quite ready for that yet. But I’ll be sure to post here when the time has come.

I’ve been mostly quiet for the last year or so, the main reason being that I’ve been either waiting anxiously for news on an exciting job (which I got) or just plain working it. BLS, as much as I like the idea of it being something that keeps food on the table, really is nothing more than my hobby at best.
That said, I haven’t had much time to try to catch up on Android development though I continue to be intrigued by it, and I’ve been slow on my web projects. There’s little excuse for it, but work has had a major impact on my activity levels at the end of the day and on weekends.


Not that I’m complaining about work…! I absolutely love my job!


That said, I’ve been trying to get back in the swing of things lately, and in particular have been working on the OAuth 2 implementation I started for Nuubz.


The first question in your mind may be “What is OAuth?” The simple explanation is that is an open standard for communicating with supporting services to allow you to register and login to websites without needing to manually create an account there. You’ve probably seen Facebook and Google login options on many websites already; OAuth is what allows that magic to work.


So why, as I’m sure your next question would be, am I implementing this myself? While there are libraries to do it, the problems are that either they’re difficult to use and/or understand, under documented, or have a license that would get in my way. I’ve tried to use a particular OAuth library I found on SourceForge several times over several years; while I got it to work somewhat, it confused the hell out of me in terms of actual usage, what data was safe to store and how to resume login sessions. The reason, besides the complexity of the library, was that it wasn’t well documented. In fact, the example code they provided literally answered nothing, not even what elements of the code were required.


While I’ve downloaded but haven’t looked at other implementations, I’ve been very hesitant too even think about using them because of licensing. I think I’ve made it clear that I hate the GPL license; I don’t want to make Nuubz open source just because I used a GPL’ed library in the software! If I decide to open source Nuubz, I want it to be because I chose to make it open source! Sure there are probably a number of OAuth libraries that are open source with a compatible license like BSD or MIT and are documented with decent examples, but I really didn’t feel like trying to hunt them down and keep them updated.


So, I decided to write my own. While I’ve been stalled for much of the last year as I mentioned above, I’ve made some important progress this week. As of this moment, my code (available at GitHub) can initiate the handshake and retrieve account information from Google. As the code is very similar for many other OAuth providers (such as Patreon, Disquss, and even Twitch)  only a few relatively minor changes are necessary to get it to work with them as well. Facebook support is coming too, though they have some additional hoops to jump through. I’m still debating Twitter… Last time I looked at implementing support for them, they didn’t provide any useful account information like email address or real name.


Now before you go off and download my code to use it, the latest changes [to make this battle station fully operational] aren’t on GitHub yet; I need to remove some debug code and clean things up a bit but I’ll have it there before this weekend ends.

So… you want to know how to upload a file via AJAX using jQuery and HTML5? Well, here’s how I’m doing it. This is not polished code, but something I cobbled together after visiting a number of sites and reading documentation in two books I have. I’m not going to provide the complete example or the PHP/server side of things, but this should be useful nonetheless.

Starting with HTML, I used the standard file input type with an onchange handler like so:

<input type="file" onchange="uploadIt(this.files);" name="uploadfile" id="uploadfile">
<div id="progressbox" >
<div id="progressbar"></div >
<div id="statustxt">0%</div>
</div>
<div id="output"</div>

The name of the field is completely arbitrary, and ultimately doesn’t matter because this field will not be submitted with your form, if you even have a form. If you do have a form, use an onsubmit() handler or your validation check to disable this field so the file isn’t uploaded a second time. By using the onchange handler, as soon as a file is chosen with the file dialog box, the upload process begins. No extra clicks are necessary. The progressbar and statustxt divisions (div) are used to indicate the upload progress, and the output division can be used for whatever output you want; while developing this code, I used it to report error messages and final status from the server.

The uploadIt function looks like this:

function uploadIt(files)
{
var file = files[0];
 switch(file.type)
 {
 case 'image/png': 
 case 'image/gif': 
 case 'image/jpeg': 
 case 'image/pjpeg':
 case 'image/webp':
 {
 $('#statustxt').html('0%');
 var fd = new FormData();
 fd.append('uploadFile',file,file.name);
 $.ajax({
 url: '/upload-handler',
 type: 'POST',
 cache: false,
 contentType: false,
 processData: false,
 data: fd,
 success: function(data, textStatus, jqXHR)
 {
 if( typeof data.error == 'undefined')
 {
 $('#output').html('<b>success!:</b> ' +data);
 }
 },
 error: function(jqXHR,textStatus,errorThrown)
 {
 $('#output').html('<b>error:</b> ' + errorThrown);
 },
 xhr: function() {
 var myXhr = $.ajaxSettings.xhr();
 if(myXhr.upload) {
 myXhr.upload.addEventListener('progress',function(event){
 var percentComplete = (event.loaded / event.total) * 100.0;
 $('#progressbox').show();
 $('#progressbar').width(percentComplete.toFixed(2) + '%') //update progressbar percent complete
 $('#statustxt').html(percentComplete.toFixed(2) + '%'); //update status text
 if(percentComplete>50)
 {
 $('#statustxt').css('color','#000'); //change status text to white after 50%
 }
 },false);
 }
 return myXhr;
 }
 });
 }
 }}

In my particular implementation, I only wanted certain file types (png, gif, jpeg, and webp as determined by the browser) to be uploaded to the server. You could also check to make sure the file is greater than 0 bytes and less than an arbitrary limit here, but I don’t have that implemented here.

Every time we have a file of the corresponding type selected, the statustxt content is set to “0%”, then we create the FormData object that will do the important job of properly formatting the file for upload. In the fd.append() call, the first parameter is the field name as it will be received on the server; this is the reason why it’s not important to have the field name set in the input tag itself. Naturally, you could copy that or use the same one, but you could just as easily set it here or make this a variable set by jQuery through another AJAX call. In PHP, this is the field name that will appear in your $_FILES variable, so you will need to know what to look for.

Next we have the file variable. While the function received an argument called “files“, and you can handle multiple files through this functionality, I wanted to maintain clarity here, and only took the first file in the files array to process and upload. Looping through the contents of files would be trivial to get this to work for multiple uploads.

The last field in fd.append() is the file name as it existed on disk at the time the file was chosen. Using this field triggers FormData to use the Content-Disposition header which, from my experience, makes the upload possible.

Next, we use jQuery to initiate the upload via the ajax function. This is where things get ugly fast, but the bottom line is you can pretty much copy and paste what I have here. But for clarity’s sake, here’s what’s what:

url is the URL to which the file will be uploaded.

type is going to be POST, though you could also use PUT in theory. There’s no reason to use PUT, but you theoretically could.

cache just as a precaution, lets make sure the browser doesn’t cache anything.

contentType is going to get set by FormData a little bit lower, so don’t manually set it. Do not confuse this with the file’s actual type or MIME type! This will get set to something akin to “multipart/form-data”.

processData we’re not going to process the data locally, so move on.

data is the FormData object, fd, that contains our file.

Next up we have 3 anonymous functions that override the success, error, and xhr handlers in the jQuery ajax call. If you don’t care about displaying progress information as the file gets uploaded, you can omit the xhr handler, though you should keep success and error so your application will know if the upload succeeded or not.

The success and error handlers here are fairly simple, they really just give you an idea of what’s happened. The success function is called if there were no errors uploading the file on the client/browser side. I repeat, the success function is called if there were no errors uploading the file on the client/browser side of things. This means that as far as the browser knows, everything went smoothly, but it’s still entirely possible that there was some error on the server, such as the server being out of space, a virus scanner might have decided the file was malware, or even just a configuration problem. Personally, I beat my head against a server side problem for a couple of hours when I couldn’t figure out why files larger than 2 MB were failing even though I had PHP configured to accept files as large as 50 MB. (Spoiler: there was a typo in a different setting in my php.ini file, which caused PHP’s ini parser to stop and use default values..)

The error function is called if there was a javascript or browser error. As stated above, this does not handle errors on the server.

The xhr handler, as presented here, is responsible for updating the progress divisions, and thus the user. xhr is shorthand for XMLHttpRequest, and we need to send jQuery a lightly prepared version of the object prior to letting it do its thing, so we’re overriding its internal instantiator for the XMLHttpRequest object. First things first, we get the copy of XMLHttpRequest this particular call is going to use through the $.ajaxSettings.xhr() call. After making sure it has the upload member, we add an event listener with another anonymous function to handle the “progress” events. With the event argument, we calculate the completion percentage percentComplete, and use that to update the status indicators.

Assuming nothing has fundamentally changed by the time you read this, your script should be able to upload a file via AJAX using jQuery and HTML5 without submitting the whole form. (Assuming you bother with one at all.)

This is a brief summary of the current development status of Nuubz.

  • All development is being done using Apache, PHP 7, PostgreSQL 9.5.x, jQuery, and Bootstrap on Linux. The software, once feature complete, will be adapted to include MySQL/MariaDB support.
  • Native account creation and login is fully functional.
  • OAuth2 support is approximately 60% implemented from scratch based on RFC 6749; obtaining access token from Google and Facebook has been tested and works properly, though more work is necessary to obtain user information from both services.
  • Support for separate read/write and read-only databases is implemented; this will allow for a master-slave/server-replicant configuration if the site administrator so desires. This will not, however, transfer files such as the comic image from server to server; a network file system is recommended for that.
  • Support for Google Analytics, reCAPTCHA, Akismet, and Project Honeypot is built in.
  • HTML5 support is the targeted HTML level.
  • Multiple language support for both interface and comic.
  • Microdata support is being implemented in the base theme.
  • Multi-home support is being implemented as well, to allow a single installation of the software to support multiple comics with different domain names. This will only require that the additional domains be parked on the server and point to the same directory.

I’m more than likely forgetting some features that have been, are being or will be implemented, but this makes a good first status report.

If you wish to play around with something, feel free to visit the comment spam tester.

Over the last few months, I’ve been [albeit slowly] working on a new piece of software. Instead of something for Android or a particular operating system, this one is for the web. While browsing my favorite webcomics, I came to realize that many were using WordPress which seemed like a problem to me. While WordPress is a hugely popular and flexible piece of software, it’s really overkill for webcomics. So, I started developing my own dedicated piece of software that I’m calling Nuubz.

It’s still relatively early in development, but if you want to watch on progress, you can visit http://dev.nuubz.com to take a look. Be sure to check back regularly as I make progress.