U.S. Hockey team lost to the Canadians... I know it's their major sport and all, but I still wanted to beat them again, especially when it counted.
On another note, this weekend I got some time to first, code with Micah and do some re-factoring. He, like my dad, does this very methodically and linearly. Its interesting because it always seems like they know exactly how any re-factoring is going to end up, no matter the complexity. This is not the case however. I am quite certain that, although sometimes they might have a feel for what it might end up looking like, or at least catch a whiff of the code smells, most of the time they just start cranking away and out pops a far cleaner chunk of code. The reason it always seems like they know what it will look like in the end is that they always know the right steps to take in order to get there.
On saturday Micah and I were looking at this beastly method I had written. I had briefly tried to re-factor it before, but when I attempted to extract out some smaller methods, they required about 6 parameters each! After reading Clean Code I know that methods are better off with fewer parameters, and if you are squeezing in more than 3, something is wrong. I decided that maybe I should be making a class or method object out of this function, but i hadn't yet gotten around to it.
When Micah looked at it, he didn't think it merited a new class, and instead tried to find good chunks to extract out. After a few moments, he decided to extract out the same chunks that I had tried. The difference was that he wasn't at all phased by all the parameters these methods would need. He motored on. He began finding simple ways to pull in the parameterized data, step by step reducing the number of inputs to the method. Once again I was a bit concerned because as we pulled more code into this method, to reduce the param count, I began noticing we were creating some duplication. But Micah motored on.
Finally, once we had reduced the input params down to 3, we looked back at the original method and extracted out another chunk of code. The same thing occurred and there were just too many inputs, except that this time I could see that not only would we be moving a lot of the parameterized data into this new method, we would also be pulling it out of the old one. Slowly but surely there was less and less duplication, and the data was being moved to where it actually belonged. Rather than calculating a huge batch of variables at the start of the method and then passing them into the algorithms, we would be calculating only the needed variables at the needed time in their most suitable location.
In the end the method was drastically simplified, something that always happens when pairing with someone like Micah, and we could finally see what was really going on.
What was important about this process was learning to persevere. I think one of the differences between and experienced coder and a novice coder is that even though neither can always see the end result, the experienced coder doesn't get worried when something gets uglier before it gets cleaner. They keep the goal in mind, and they take the familiar steps that usually lead to a successful result.
This weekend I also had some time to progress further with the Rails book. One basic step I thought was important is what to do when starting a project. It seems to me that you have 3 options when starting a new project. You can either jump right into it and code without any planning, you can try to plan it from start to finish with a detailed process, or you can just sketch out some rough ideas about how you think it might all work. Experts in the field seem to suggest that you should just make a rough sketch of what you are doing since one, it is useful to have a picture for both you and the customer, and two, because projects are always going to change from what you originally imagined.
The Rails book suggests that you make quick sketches with paper and a pencil to layout the feel of the project in a very broad sense. This will help you see eye to eye with your customer, and it is easy to throw away.
Time to get back to coding another rails example project!
Sunday, February 28, 2010
Friday, February 26, 2010
Day 37
I apologize for being late, last night I got caught up in my Sociology hw and I completely forgot to blog. Yesterday we committed out work for the completed feature, and it got deployed in the real site. Pretty cool how fast we were able to add a new feature, and then see it already in place on their website. We actually got the feature done so fast that they just weren't ready for it, which I believe is what Micah was aiming for. Suddenly this thing that they thought would take a couple of weeks to finish was done in three days, and they were completely surprised.
This of course was the good sort of surprise, and they quite quickly brought in one of the people who would be using the feature often so that they could see how it would work. Unfortunately I had to go to class right as Micah was demoing it, but it went well and apparently they were so impressed that they included it in their deployment right away!
The client does continuous deployment, and each week they push up to the main server, so every time anyone commits code they have to make sure everything still works. It seems kind of scary, since you could potentially deploy a secretly bugged build up to the live site, but of course you can always roll back if that turns out to be the case. It does, however, let them very rapidly develop their site and there is always something new for people use and take advantage of.
Some interesting things I had never though of occurred when we were setting up the error messages that the user might see. First off, since typically the business logic belongs in the models, we wanted to be passing the error message from the model through the controller. The client had, however, been using generic error messages located in the controller. If some request failed, they would check to see if the model had populated an object with any error messages, and if there was one, they would print out a generic. We wanted to print out a more informative and meaningful error, but here is the rub - if you are making an international app, you want to have as few error messages and such as you can so that translating all of them is a straight forward process.
This of course was the good sort of surprise, and they quite quickly brought in one of the people who would be using the feature often so that they could see how it would work. Unfortunately I had to go to class right as Micah was demoing it, but it went well and apparently they were so impressed that they included it in their deployment right away!
The client does continuous deployment, and each week they push up to the main server, so every time anyone commits code they have to make sure everything still works. It seems kind of scary, since you could potentially deploy a secretly bugged build up to the live site, but of course you can always roll back if that turns out to be the case. It does, however, let them very rapidly develop their site and there is always something new for people use and take advantage of.
Some interesting things I had never though of occurred when we were setting up the error messages that the user might see. First off, since typically the business logic belongs in the models, we wanted to be passing the error message from the model through the controller. The client had, however, been using generic error messages located in the controller. If some request failed, they would check to see if the model had populated an object with any error messages, and if there was one, they would print out a generic. We wanted to print out a more informative and meaningful error, but here is the rub - if you are making an international app, you want to have as few error messages and such as you can so that translating all of them is a straight forward process.
Wednesday, February 24, 2010
Day 36
Diving deeper into Rails. There is a framework used by our client which is both fairly impressive, and quite complicated. It is powerful in that it handles mostly all of the needed CRUD for controllers, with a very polymorphic and expansive system, but it is also enormously complicated an difficult to understand. Diving into the code certainly helps, but it sacrifices clarity for density. None the less, seeing the bare bones for a new framework which, once understood, can drastically increase your productivity, is pretty cool. I wish I had been there to see its development from birth.
Working more with Micah today, we got a lot done on our current feature. I revamped my Cucumber knowledge, getting to write a few step definitions and acceptance tests recently. Cucumber reads quite naturally and is pretty easy to work with.
I also learned more about the REST convention, and following it conventions for URLs and routing. This, along with defining a separation between all the dynamic structure of a page (like the Java Script) and the actual html for the page. The client, in the spirit of being Restful (btw, Rails 3.0 is going to incorporate this by default), likes to keep a web page as pure html and thus have all the JS in header operations before the page's html is constructed. In this way, if a browser doesn't accept JS, it can still read the page, its just that requests that require JS wont do anything.
Got another 30 pages done with the Rails book as well. At this point, mostly everything learned is learned through example and practice, thus I am writing a lot of code while reading. Naturally, I am making the pet project that the book instructs you to make, but along side that I am going to work on my personal website so that I can really put what I learn into practice. It is very important that I design a Rails App from start to finish on my own so that I can develop a better understanding of the separation between the client's code and built in Rails features.
I find that it is far more important to understand what is going on, than to merely know what is going on. The first is far more challenging, but also much more powerful. It is just like highschool and college physics. You can memorize all the formulas and equations you want, but come test time if you don't understand the material, you can still get caught up by a tough word problem. Understanding gives you even more than this, since if you understand the fundamentals of a formula or a problem, then you can recreate the needed equations on the spot without having them memorized. The same is true with coding, algorithms, and syntax. You can memorize and know all you want, but until you understand what you are working with, you wont be able to design creative solutions.
Sorry for being brief, I have some Sociology hw to do, and I NEED sleep! Early morning trips to Chicago are taking their toll
Working more with Micah today, we got a lot done on our current feature. I revamped my Cucumber knowledge, getting to write a few step definitions and acceptance tests recently. Cucumber reads quite naturally and is pretty easy to work with.
I also learned more about the REST convention, and following it conventions for URLs and routing. This, along with defining a separation between all the dynamic structure of a page (like the Java Script) and the actual html for the page. The client, in the spirit of being Restful (btw, Rails 3.0 is going to incorporate this by default), likes to keep a web page as pure html and thus have all the JS in header operations before the page's html is constructed. In this way, if a browser doesn't accept JS, it can still read the page, its just that requests that require JS wont do anything.
Got another 30 pages done with the Rails book as well. At this point, mostly everything learned is learned through example and practice, thus I am writing a lot of code while reading. Naturally, I am making the pet project that the book instructs you to make, but along side that I am going to work on my personal website so that I can really put what I learn into practice. It is very important that I design a Rails App from start to finish on my own so that I can develop a better understanding of the separation between the client's code and built in Rails features.
I find that it is far more important to understand what is going on, than to merely know what is going on. The first is far more challenging, but also much more powerful. It is just like highschool and college physics. You can memorize all the formulas and equations you want, but come test time if you don't understand the material, you can still get caught up by a tough word problem. Understanding gives you even more than this, since if you understand the fundamentals of a formula or a problem, then you can recreate the needed equations on the spot without having them memorized. The same is true with coding, algorithms, and syntax. You can memorize and know all you want, but until you understand what you are working with, you wont be able to design creative solutions.
Sorry for being brief, I have some Sociology hw to do, and I NEED sleep! Early morning trips to Chicago are taking their toll
Tuesday, February 23, 2010
Day 35
Another excellent day in Chicago. Today I got to meet the CEO and founder of the current client. Our introduction was rather brief and, as one would expect, we got straight to business talking about the details of the first feature we are making for them.
One of the more interesting issues tackled today was a race condition where several users are trying to access the same expiring option at the same time. Say we have this sweepstakes going on, and only 10 people get to sign up for. This sweepstakes is released on twitter, and immediately 50 people apply to sign up, all at virtually the same time. You have a database which holds only ten redeemable account numbers, but this database is being accessed by several servers at the same time! On the very last redeemable account, 5 people request the server at such a close interval that each of their servers request the database for the same account at the same time. You can say that the last account should go to the first request for it, but what these requests are being done in parallel, so the first request in might be the last request out. Who gets it? How do you check too see if there is still an account remaining while other servers are grabbing the last account?
Turns out that Ruby with MySQL has a very nice lock! method that can be called in a model to guarantee only one write can be made to a certain row in the database at any given time. It does something else, but let me query you first. Is that enough?
Even if only one of these 5 servers was able to write to this database row, thus editing it to grabbing the last account, the 4 other servers could still read the row and therefore determine (if reading just before the row was being edited, but after it was locked) there is still an account left. The remaining servers might then try to edit the row the moment the lock was released, which could cause some issues.
Its not enough to simply lock the writing of this row, but you also must lock it so that nothing else can be reading it while it is being written to. In this way, the first of the 5 servers would lock the row, at which point the other 4 servers would have to wait until the 5th server officially had the last account before they could read the row.
Fortunately this is what the lock! method does, when using the LOCK IN SHARE MODE. This along with using a Ruby transaction (which is an all or nothing access for an object), quite nicely resolved this race condition.
Micah wrote up a script to test this (because unfortunately rspec performs its tests using transactions rather than actually accessing the database, thus making testing this race condition virtually impossible in the typical form) which would create 5 Ruby forks (not threads since Ruby threads all run through the same interpreter, and thus are really actually just one thread [ Not in JRuby though!!]) which would all request the last redeemable account at the same time. We confirmed that it worked by seeing a couple of forks grabbing the last account while only using Ruby transactions, and then confirmed the lock! worked by running this script again and again. Micah actually did a cute little command to run the script 30 times, and it never failed even through that. Though it seems nearly impossible to be certain that it worked, it seems quite likely that it works.
Start reading the Agile Web Development With Rails - Third Edition, and got about 30 pages in tonight. I am really glad I started reading this after creating my Socket Server because not only does it solidify my server knowledge, it also gives me a new realm of connections to make so that I better understand how Rails really works.
A quick overview of what I have gone through so far.
Chapter 1 was mostly all intros and talk about how Rails have developed and how this book will be structured, so never mind that. Chapter 2 is where the details begin.
Chapter 2 revolves around explaining MVC.
MVC is of course the Model View Controller framework that Rails, along with many many other applications, uses.
2.1
MVC - Model View Controller
Model - handles all data, data interaction, and business rules. Gate keeper and data store.
View - UI, normally based on model. Never handles incoming data. Different views for same model.
Controllers - The corner stone of application. Receive requests, handles them using the models, display results using views.
Handling requests:
http://my.url/store/add_to_cart/123
store - name of controller
add_to_cart - name of action
123 - extracted into id
The controller find's the user shopping cart and product (model)
Tells the cart to add the product
Shows updated info to user
2.2
Models--
Organizing data (ObjectOriented vs relation databases):
Database-centric - Bringing data and database modifications directly into production code. Ewwwwww
Object-Relational Mapping - Uses OO to control data. Maps Classes to tables, objects to rows, and columns to attributes.
Active Record - Follows the ORM model.
2.3
Views --
Often just batches of HTML, but typically wish to have some actions. Actions provided by action methods in the controllers.
Dynamic content done in 3 ways using templates:
ERb (embedded ruby)
XML Builder
RJS (JavaScript) - Ajax
Controllers --
Responsibilities - Routing external requests (friendly URLs), Managing Caching, Managing helper modules (for views), Managing sessions
Certainly a quick summary of the chapter, but if you want to know more details, the book itself it a pretty smooth read while retaining high content.
One of the more interesting issues tackled today was a race condition where several users are trying to access the same expiring option at the same time. Say we have this sweepstakes going on, and only 10 people get to sign up for. This sweepstakes is released on twitter, and immediately 50 people apply to sign up, all at virtually the same time. You have a database which holds only ten redeemable account numbers, but this database is being accessed by several servers at the same time! On the very last redeemable account, 5 people request the server at such a close interval that each of their servers request the database for the same account at the same time. You can say that the last account should go to the first request for it, but what these requests are being done in parallel, so the first request in might be the last request out. Who gets it? How do you check too see if there is still an account remaining while other servers are grabbing the last account?
Turns out that Ruby with MySQL has a very nice lock! method that can be called in a model to guarantee only one write can be made to a certain row in the database at any given time. It does something else, but let me query you first. Is that enough?
Even if only one of these 5 servers was able to write to this database row, thus editing it to grabbing the last account, the 4 other servers could still read the row and therefore determine (if reading just before the row was being edited, but after it was locked) there is still an account left. The remaining servers might then try to edit the row the moment the lock was released, which could cause some issues.
Its not enough to simply lock the writing of this row, but you also must lock it so that nothing else can be reading it while it is being written to. In this way, the first of the 5 servers would lock the row, at which point the other 4 servers would have to wait until the 5th server officially had the last account before they could read the row.
Fortunately this is what the lock! method does, when using the LOCK IN SHARE MODE. This along with using a Ruby transaction (which is an all or nothing access for an object), quite nicely resolved this race condition.
Micah wrote up a script to test this (because unfortunately rspec performs its tests using transactions rather than actually accessing the database, thus making testing this race condition virtually impossible in the typical form) which would create 5 Ruby forks (not threads since Ruby threads all run through the same interpreter, and thus are really actually just one thread [ Not in JRuby though!!]) which would all request the last redeemable account at the same time. We confirmed that it worked by seeing a couple of forks grabbing the last account while only using Ruby transactions, and then confirmed the lock! worked by running this script again and again. Micah actually did a cute little command to run the script 30 times, and it never failed even through that. Though it seems nearly impossible to be certain that it worked, it seems quite likely that it works.
Start reading the Agile Web Development With Rails - Third Edition, and got about 30 pages in tonight. I am really glad I started reading this after creating my Socket Server because not only does it solidify my server knowledge, it also gives me a new realm of connections to make so that I better understand how Rails really works.
A quick overview of what I have gone through so far.
Chapter 1 was mostly all intros and talk about how Rails have developed and how this book will be structured, so never mind that. Chapter 2 is where the details begin.
Chapter 2 revolves around explaining MVC.
MVC is of course the Model View Controller framework that Rails, along with many many other applications, uses.
2.1
MVC - Model View Controller
Model - handles all data, data interaction, and business rules. Gate keeper and data store.
View - UI, normally based on model. Never handles incoming data. Different views for same model.
Controllers - The corner stone of application. Receive requests, handles them using the models, display results using views.
Handling requests:
http://my.url/store/add_to_cart/123
store - name of controller
add_to_cart - name of action
123 - extracted into id
The controller find's the user shopping cart and product (model)
Tells the cart to add the product
Shows updated info to user
2.2
Models--
Organizing data (ObjectOriented vs relation databases):
Database-centric - Bringing data and database modifications directly into production code. Ewwwwww
Object-Relational Mapping - Uses OO to control data. Maps Classes to tables, objects to rows, and columns to attributes.
Active Record - Follows the ORM model.
2.3
Views --
Often just batches of HTML, but typically wish to have some actions. Actions provided by action methods in the controllers.
Dynamic content done in 3 ways using templates:
ERb (embedded ruby)
XML Builder
RJS (JavaScript) - Ajax
Controllers --
Responsibilities - Routing external requests (friendly URLs), Managing Caching, Managing helper modules (for views), Managing sessions
Certainly a quick summary of the chapter, but if you want to know more details, the book itself it a pretty smooth read while retaining high content.
Monday, February 22, 2010
Day 34
Today we were at their office in Chicago meeting all of the guys of the new Client, getting the source code up to speed on our computers, and learning what we will be doing for them. I must say, a pretty amazing company. Their relaxed, but hard working on target mindset is extremely refreshing and conducive to an effective working environment. Their website has been wildly successful and is saving tons of people tons of money, while bringing companies tons of customers and tons of money. Where's the fault? It actually comes in when certain companies start saying "You need to set a limit! Too many customers!" Ayyy, there's the rub.
Getting started on a new existing Rail's project is always a pain. Even more so than Java or plain Ruby projects. On every Rail's project there is a new set of 10-20 gems and plugins that have to be settled. Each gem with their own specific version, frequently not the most up to date version either. On top of this, there is a wide selection of databases that can be used, and always require some love and finagling to make them happy.
I spent around 4-5 hours today getting my situation up to date and on track with their environment. Micah was able to get going in about 2 1/2 or so hours, so he got a good look at the code and a fair feel for what we will have to do to complete the feature. On a happier note though, the guys here said we got up and ready faster than anyone else before us, who often took up to a week!
Most of my time was spent getting MySQL working. I jump from error to error, then fixed stuff, but went back to the same error as before. It mostly all stemmed back to not being on Snow Leopard, but the default download for mysql was using the x86_64 archive. Later on I figured this out, went back and got the right version - i386 archive, but still had issues with missing refs.
One really useful tool I learned about is the RVM. Ruby Version Manager. It can switch back and forth between different Ruby environments, with different versions of Ruby (like the core Ruby or Ruby Enterprise Edition (which is mega faster)), different gems and gem versions. With just a simple command like: rvm ree I could switch to Ruby Enterprise Edition with all the gems I needed for this client, and then with: rvm system I could swap back to my normal Ruby environment. Uber powerful.
Tomorrow we will start planning out our iterations and our story cards.
Getting started on a new existing Rail's project is always a pain. Even more so than Java or plain Ruby projects. On every Rail's project there is a new set of 10-20 gems and plugins that have to be settled. Each gem with their own specific version, frequently not the most up to date version either. On top of this, there is a wide selection of databases that can be used, and always require some love and finagling to make them happy.
I spent around 4-5 hours today getting my situation up to date and on track with their environment. Micah was able to get going in about 2 1/2 or so hours, so he got a good look at the code and a fair feel for what we will have to do to complete the feature. On a happier note though, the guys here said we got up and ready faster than anyone else before us, who often took up to a week!
Most of my time was spent getting MySQL working. I jump from error to error, then fixed stuff, but went back to the same error as before. It mostly all stemmed back to not being on Snow Leopard, but the default download for mysql was using the x86_64 archive. Later on I figured this out, went back and got the right version - i386 archive, but still had issues with missing refs.
One really useful tool I learned about is the RVM. Ruby Version Manager. It can switch back and forth between different Ruby environments, with different versions of Ruby (like the core Ruby or Ruby Enterprise Edition (which is mega faster)), different gems and gem versions. With just a simple command like: rvm ree I could switch to Ruby Enterprise Edition with all the gems I needed for this client, and then with: rvm system I could swap back to my normal Ruby environment. Uber powerful.
Tomorrow we will start planning out our iterations and our story cards.
Sunday, February 21, 2010
Day 33
I finished the HTTP server on time for my demo on Friday, which went smoothly. This week Micah and I will be in Chicago doing some work for a web based company. Their website hosts coupons for businesses. A business, say Gino's East will post a possible coupon on their website for maybe 50% off a large pizza, and if 300 people sign up for the coupon, then Gino's will carry through with it. Anyone who wants the coupon can sign up for it on the day it is hosted, but tomorrow brings a new coupon and the end to the sales of the old one. They have been very successful, and their site is pretty awesome.
Ok, I will just share some of the basic tid-bits of how the server worked.
I made a very very basic site for the server to host in order to test to make sure it worked. This site had certain requirements which my server had to support. It had to have a few basic pages (like the initial page '/index.html and a homepage), catch incorrect URL requests and respond with a 404, support common MIME types (images, pdf, even small video files), have an Echo page which would spit back out the arguments past to it via the URL, and handle several requests at the same time.
Hosting the basic pages was fairly straight forward once you know how to format a request and response. The most basic request header looked like this:
GET /index.html HTTP/1.1
Host: localhost:80
The form is Request type page requested protocol
Host: resource name : port
The resource name could be localhost or could ben something like www.google.com.
The most basic response header would look like this:
HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: 500
The form is Protocol response code and response type (for an error page the code would be 404 and type Not Found)
Typically there is a flurry of other information like cookie and browser information. After the header would follow the body of the response.
For my home page, and like I said... super basic pages, the response body would just be some html like
<%html><%h1>This is the home page!
Sweet website right?
But then for a page with an image or pdf there were a lot of extra steps. First, in the response header you have to figure out what the page's MIME type will be, and then you have to get the size of the page. An image would be like image/jpeg or image/gif, and then you send the size viz the Content-Length param.
Once you ship the header, you have to send the response body, which will contain the image file. TCP/IP sends all files in little chunks, so you can't just ship the whole image at once. You have to break it up into individual bytes and stream it onto an output stream. This was a bit of a pain at first because even with some of the built in Java support for these IO streams, it can be a bit tricky figuring out which types to use, and how to make all the needed conversions.
For example, to send an image as a byte stream given only its location there are several steps you need to take. First, in Java, you need to create a File object for the image. For this, you need a File Output Stream to read the file and put it into local memory. Then you need to turn that file into a byte stream, and get its size. Finally you write that byte stream into a Byte Array Output Stream, which can then ship the bytes onto your Socket's output stream.
A fairly lengthy process to merely ship photo's from computer to computer, but it is reliable and will work with virtually any file type.
At last, making an echo page is pretty simple. When you receive a request with some input params, they are tide into the URL something like this:
www.justinmmartin.com:80/index.html?a=1&b=2&c=3
where a,b,c are the params. The request has the params in the initial GET line:
GET /index.html?a=1&b=2&c=3 HTTP/1.1
So you can just parse out the string, snatch up the params by splitting the string on certain characters, and then add them back into HTML to be shipped out in your response.
A relatively broad summary for what needs to be done to create a basic server, but hopefully you get the picture.
Ok, I will just share some of the basic tid-bits of how the server worked.
I made a very very basic site for the server to host in order to test to make sure it worked. This site had certain requirements which my server had to support. It had to have a few basic pages (like the initial page '/index.html and a homepage), catch incorrect URL requests and respond with a 404, support common MIME types (images, pdf, even small video files), have an Echo page which would spit back out the arguments past to it via the URL, and handle several requests at the same time.
Hosting the basic pages was fairly straight forward once you know how to format a request and response. The most basic request header looked like this:
GET /index.html HTTP/1.1
Host: localhost:80
The form is Request type page requested protocol
Host: resource name : port
The resource name could be localhost or could ben something like www.google.com.
The most basic response header would look like this:
HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: 500
The form is Protocol response code and response type (for an error page the code would be 404 and type Not Found)
Typically there is a flurry of other information like cookie and browser information. After the header would follow the body of the response.
For my home page, and like I said... super basic pages, the response body would just be some html like
<%html><%h1>This is the home page!
Sweet website right?
But then for a page with an image or pdf there were a lot of extra steps. First, in the response header you have to figure out what the page's MIME type will be, and then you have to get the size of the page. An image would be like image/jpeg or image/gif, and then you send the size viz the Content-Length param.
Once you ship the header, you have to send the response body, which will contain the image file. TCP/IP sends all files in little chunks, so you can't just ship the whole image at once. You have to break it up into individual bytes and stream it onto an output stream. This was a bit of a pain at first because even with some of the built in Java support for these IO streams, it can be a bit tricky figuring out which types to use, and how to make all the needed conversions.
For example, to send an image as a byte stream given only its location there are several steps you need to take. First, in Java, you need to create a File object for the image. For this, you need a File Output Stream to read the file and put it into local memory. Then you need to turn that file into a byte stream, and get its size. Finally you write that byte stream into a Byte Array Output Stream, which can then ship the bytes onto your Socket's output stream.
A fairly lengthy process to merely ship photo's from computer to computer, but it is reliable and will work with virtually any file type.
At last, making an echo page is pretty simple. When you receive a request with some input params, they are tide into the URL something like this:
www.justinmmartin.com:80/index.html?a=1&b=2&c=3
where a,b,c are the params. The request has the params in the initial GET line:
GET /index.html?a=1&b=2&c=3 HTTP/1.1
So you can just parse out the string, snatch up the params by splitting the string on certain characters, and then add them back into HTML to be shipped out in your response.
A relatively broad summary for what needs to be done to create a basic server, but hopefully you get the picture.
Friday, February 19, 2010
Day 32
I learned a LOT today... well yesterday, last night, and this morning. Basically, the last few days I learned all about how to make a Socket Server and how HTTP requests work, but I really didn't know how to make them happen. Today, I learned that. There is a massive difference between understanding the principles behind how something should work, and the understanding required to actually make that something work.
Today I learned how an HTTP server, and all the communication involved in the process, works - in practice.
I learned how to form real, and usable requests and response.
I learned how to use them to display different types of html pages.
I learned how to link them to a file system, such that a request routes directly to a directory, and from it, derives the appropriate response.
I learned how to run multiple connections.
I re-learned how to make a build.xml file to run an ant process so that I can snatch out a .jar file.
I learned about pretty much every type of input and output stream Java has to offer, as well has how to link them up in unison to move files around in efficient manners.
I learned how to use chunked response, although not in practice... But i am pretty sure I could put it into practice in a reasonable amount of time.
I learned how to send different MIME types across the web so that I can display images and such from a local file onto a browser.
And I learned a whole lot of other stuff in order to learn how to learn what I needed to learn! This is likely the most important thing I derived from the long day of hardship.
Here is the rub... its now like 4:15 AM and I am not quite finished implementing all the mime types (though pretty close) or being able to pick up parameters (another fairly short matter). I gotta have it done by about noon tomorrow so... Imma sleep and blog more this weekend when my brain can recollect details.
P.S. telnet is pretty sweet terminal command. check it out
Today I learned how an HTTP server, and all the communication involved in the process, works - in practice.
I learned how to form real, and usable requests and response.
I learned how to use them to display different types of html pages.
I learned how to link them to a file system, such that a request routes directly to a directory, and from it, derives the appropriate response.
I learned how to run multiple connections.
I re-learned how to make a build.xml file to run an ant process so that I can snatch out a .jar file.
I learned about pretty much every type of input and output stream Java has to offer, as well has how to link them up in unison to move files around in efficient manners.
I learned how to use chunked response, although not in practice... But i am pretty sure I could put it into practice in a reasonable amount of time.
I learned how to send different MIME types across the web so that I can display images and such from a local file onto a browser.
And I learned a whole lot of other stuff in order to learn how to learn what I needed to learn! This is likely the most important thing I derived from the long day of hardship.
Here is the rub... its now like 4:15 AM and I am not quite finished implementing all the mime types (though pretty close) or being able to pick up parameters (another fairly short matter). I gotta have it done by about noon tomorrow so... Imma sleep and blog more this weekend when my brain can recollect details.
P.S. telnet is pretty sweet terminal command. check it out
Wednesday, February 17, 2010
Day 31
I am just going to title this post because I know the connotation it will inevitably invoke.
How the SocketServer services the Client's Socket
I apologize if you find this unnecessary or inappropriate, but I can only imagine what is going through Kichu's head while he reads this.
Creating a multi-threaded system is always interesting. It is extremely difficult to be certain none of your threads will collide. It is virtually impossible to be certain if you have no tests. There are always a multitude of unexpected errors, race conditions, and concurrent modification issues (yes yes, I know this is a java error name) that can pop out and surprise you... or worse, lurk around until your not paying attention and then bite you in the a**. This is why it is important to be consistently coding in a thread safe manner, using locks and semaphores when you can, testing lots of possible variations of how the code can be run, and running your tests several times in a row each time you make a threading change.
For my Socket Server I had to spawn up a flurry of threads to run many simultaneous actions. So first, a brief over view of how the Socket Server worked, and then I will explain how I remained thread safe.
Some application would start up the Socket Server, which would create a Server Socket to watch a specific port. Then presumably this application would like to be able to accomplish others tasks while the Server Socket is simultaneously monitoring its specific port, thus the Server Socket needs to sit in its own thread called the ServerSocketThread.
The Server Socket is in a constant loop, checking to see if any Client Sockets have tried to talk on the specified port, for as long as the Server Socket is let to be open. If a Client Socket attempts to connect on this port, the Server Socket needs to accept the client's request. To accept, the Server Socket must create a server side socket to pair with the Client Socket, and then instruct this server side socket, called the SocketServicer, to serve the Client Socket. The SocketServicer has to serve the Client Socket while the Server Socket is still monitoring the port, thus a new thread, called a nobleServiceThread, must be created for every single Client Socket- SocketServicer pair.
Once the SocketServicer has finished serving the Client Socket, the nobleServiceThread will automatically close itself and end the connection as well as terminate itself.
If the Server wants to close one of it's Server Sockets, the Server Socket will have to make sure all of it's SocketServicers and nobleServiceThreads are closed and terminated before it can be closed. There must, therefore, be a list of all of the nobleServiceThreads. The Server Socket must loop through this nobleServiceThreads list and terminate each thread. To simply kill an active Socket, however, would mean you are suddenly cutting a client off in the middle of an interaction, and that is bad practice. Thus the nobleServiceThreads must first be given a chance for the Socket to finish and close up before they are cut off, and as such must be given a TimeOut Period before they are automatically cut off regardless of status. Once cut off, the deactive thread must be removed from the list.
Ok, now down to business.
There are a few smaller and somewhat trivial threading issues, some of which involving testing, but I am just going to go over the biggest and most interesting one I encountered.
So we have this list of nobleServiceThreads that each are running a Socket which can finish at any time. We also have a Server Socket which might want to close at any time, thus having to close all of the nobleServiceThreads. Thus, right away we can see the race condition. Can you guess what it is?
If, while looping through and closing the list of nobleServiceThreads, a Socket finishes it's interactions and decides to close, it could potentially remove itself from the list we are iterating through. This means we could be trying to remove a thread that is at the same time removing itself. Lets look at some code:
First, here is the Server Socket's thread/loop which watches a port and accepts incoming Client Sockets:
We can see that as long as the Server Socket is open, it will continue to make new threads for incoming Clients.
Here is what the nobleServiceThread does:
I call them nobleServiceThreads because if they are about to be killed, their very last wish is to remove themselves from the list such that they don't burden the ServerSocket with dead weight. How kind.
The first step to preventing the thread from removing itself from the list while it is being removed elsewhere, is to make the nobleServiceThreads list a Synchronized list:
Next, since we don't want a thread to be potentially modifying our list while we are iterating through it, we don't use an iterator to go through the list. Instead we just loop while the list isn't empty:
Now this is pretty good, but we still have an issue. This one is a little tougher to guess. If, and this actually happened about 1/5 times when I ran the tests, a nobleServiceThread is active right as the while (nobleServiceThreads.size() > 0) is checked, and then right before the very next line, the nobleServiceThread removes itself from the list - then the nobleServiceThreads.get(0) could return null (if this was the very last thread).
This is a tough issue. For awhile I was almost tempted to put in 100 if statements checking to see if the thread was still there before I removed it, but of course that wouldn't change much. So instead, with Micah's help, here is the solution I've got:
The mutex is just a regular object that the synchronized block can hold onto, thus preventing anything else, trying to use the same mutex in a different synchronized block, from acting. Here is how the nobleServiceThread changed:
This prevents the nobleServiceThread from removing itself from the list if the ServerSocket is looking at this same nobleServiceThread to remove it.
Now you might ask " Since the nobleServiceThread's lastBreath is used in the finally{ ... } clause (which will execute in the end, no matter what, when you try to kill the thread), why not just call interrupt on the thread as soon as you've got it?"
This is because the applicationServer.serve() method might be talking to a client. If it is waiting on a response from the client, and thus is in a reading block, than the interrupt wont reach the thread until the reading block is finished. Thus, the interrupt could potentially wait forever while the thread waits for a response.
The interrupt signal actually only reaches the thread in 3 instances. When the thread is in a wait(), join(), or sleep() method. So a thread can only be interrupted when it isn't doing anything. (In case you didn't know, a thread is in wait() while it is waiting for its chance to use the processor, and in a join() when the application is telling it to finish its last task and then terminate). So if you send an interrupt signal to a thread while it is performing some task, a flag will be marked and next time the thread is waiting, the JVM will check the flag and terminate the thread.
I must give credit to the Software Craftsmanship Articles 8-11 written by Uncle Bob, since they walked through many of the same steps I listen here.
How the SocketServer services the Client's Socket
I apologize if you find this unnecessary or inappropriate, but I can only imagine what is going through Kichu's head while he reads this.
Creating a multi-threaded system is always interesting. It is extremely difficult to be certain none of your threads will collide. It is virtually impossible to be certain if you have no tests. There are always a multitude of unexpected errors, race conditions, and concurrent modification issues (yes yes, I know this is a java error name) that can pop out and surprise you... or worse, lurk around until your not paying attention and then bite you in the a**. This is why it is important to be consistently coding in a thread safe manner, using locks and semaphores when you can, testing lots of possible variations of how the code can be run, and running your tests several times in a row each time you make a threading change.
For my Socket Server I had to spawn up a flurry of threads to run many simultaneous actions. So first, a brief over view of how the Socket Server worked, and then I will explain how I remained thread safe.
Some application would start up the Socket Server, which would create a Server Socket to watch a specific port. Then presumably this application would like to be able to accomplish others tasks while the Server Socket is simultaneously monitoring its specific port, thus the Server Socket needs to sit in its own thread called the ServerSocketThread.
The Server Socket is in a constant loop, checking to see if any Client Sockets have tried to talk on the specified port, for as long as the Server Socket is let to be open. If a Client Socket attempts to connect on this port, the Server Socket needs to accept the client's request. To accept, the Server Socket must create a server side socket to pair with the Client Socket, and then instruct this server side socket, called the SocketServicer, to serve the Client Socket. The SocketServicer has to serve the Client Socket while the Server Socket is still monitoring the port, thus a new thread, called a nobleServiceThread, must be created for every single Client Socket- SocketServicer pair.
Once the SocketServicer has finished serving the Client Socket, the nobleServiceThread will automatically close itself and end the connection as well as terminate itself.
If the Server wants to close one of it's Server Sockets, the Server Socket will have to make sure all of it's SocketServicers and nobleServiceThreads are closed and terminated before it can be closed. There must, therefore, be a list of all of the nobleServiceThreads. The Server Socket must loop through this nobleServiceThreads list and terminate each thread. To simply kill an active Socket, however, would mean you are suddenly cutting a client off in the middle of an interaction, and that is bad practice. Thus the nobleServiceThreads must first be given a chance for the Socket to finish and close up before they are cut off, and as such must be given a TimeOut Period before they are automatically cut off regardless of status. Once cut off, the deactive thread must be removed from the list.
Ok, now down to business.
There are a few smaller and somewhat trivial threading issues, some of which involving testing, but I am just going to go over the biggest and most interesting one I encountered.
So we have this list of nobleServiceThreads that each are running a Socket which can finish at any time. We also have a Server Socket which might want to close at any time, thus having to close all of the nobleServiceThreads. Thus, right away we can see the race condition. Can you guess what it is?
If, while looping through and closing the list of nobleServiceThreads, a Socket finishes it's interactions and decides to close, it could potentially remove itself from the list we are iterating through. This means we could be trying to remove a thread that is at the same time removing itself. Lets look at some code:
First, here is the Server Socket's thread/loop which watches a port and accepts incoming Client Sockets:
private Thread makeSocketThread()
{
return new Thread(new Runnable()
{
public void run()
{
while (serverSocketOpen)
{
runServerSocket();
}
}
});
}
private void runServerSocket()
{
try
{
Socket clientSocket = serverSocket.accept();
Thread servicerThread = new Thread(new ServiceRunner(clientSocket));
nobleServiceThreads.add(servicerThread);
servicerThread.start();
}
catch (IOException e)
{
}
}
We can see that as long as the Server Socket is open, it will continue to make new threads for incoming Clients.
Here is what the nobleServiceThread does:
private class ServiceRunner implements Runnable
{
Socket clientSocket;
public ServiceRunner(Socket clientSocket)
{
this.clientSocket = clientSocket;
}
public void run()
{
try
{
applicationServer.serve(clientSocket);
clientSocket.close();
}
catch (IOException e)
{
e.printStackTrace();
}
finally{
lastBreath();
}
}
private void lastBreath()
{
nobleServiceThreads.remove(Thread.currentThread());
}
}
I call them nobleServiceThreads because if they are about to be killed, their very last wish is to remove themselves from the list such that they don't burden the ServerSocket with dead weight. How kind.
The first step to preventing the thread from removing itself from the list while it is being removed elsewhere, is to make the nobleServiceThreads list a Synchronized list:
private List<Thread> nobleServiceThreads = Collections.synchronizedList(new ArrayList<Thread>());
Next, since we don't want a thread to be potentially modifying our list while we are iterating through it, we don't use an iterator to go through the list. Instead we just loop while the list isn't empty:
public void close() throws IOException, InterruptedException
{
if (serverSocketOpen)
{
serverSocketOpen = false;
while (nobleServiceThreads.size() > 0)
{
nobleServiceThreads.get(0).join();
nobleServiceThreads.remove(0);
}
serverSocket.close();
}
else
serverSocket.close();
}
Now this is pretty good, but we still have an issue. This one is a little tougher to guess. If, and this actually happened about 1/5 times when I ran the tests, a nobleServiceThread is active right as the while (nobleServiceThreads.size() > 0) is checked, and then right before the very next line, the nobleServiceThread removes itself from the list - then the nobleServiceThreads.get(0) could return null (if this was the very last thread).
This is a tough issue. For awhile I was almost tempted to put in 100 if statements checking to see if the thread was still there before I removed it, but of course that wouldn't change much. So instead, with Micah's help, here is the solution I've got:
public void close() throws IOException, InterruptedException
{
if (serverSocketOpen)
{
serverSocketOpen = false;
serverSocket.close();
serverSocketThread.join();
while (nobleServiceThreads.size() > 0)
{
Thread thread = null;
synchronized (mutex)
{
if (nobleServiceThreads.size() > 0)
{
thread = nobleServiceThreads.get(0);
}
}
if (thread != null)
{
thread.join(TIMEOUT_PERIOD);
synchronized (mutex)
{
if (nobleServiceThreads.contains(thread))
{
nobleServiceThreads.remove(0);
}
}
}
}
}
else
serverSocket.close();
}
The mutex is just a regular object that the synchronized block can hold onto, thus preventing anything else, trying to use the same mutex in a different synchronized block, from acting. Here is how the nobleServiceThread changed:
private void lastBreath()
{
synchronized (mutex)
{
nobleServiceThreads.remove(Thread.currentThread());
}
}
This prevents the nobleServiceThread from removing itself from the list if the ServerSocket is looking at this same nobleServiceThread to remove it.
Now you might ask " Since the nobleServiceThread's lastBreath is used in the finally{ ... } clause (which will execute in the end, no matter what, when you try to kill the thread), why not just call interrupt on the thread as soon as you've got it?"
This is because the applicationServer.serve() method might be talking to a client. If it is waiting on a response from the client, and thus is in a reading block, than the interrupt wont reach the thread until the reading block is finished. Thus, the interrupt could potentially wait forever while the thread waits for a response.
The interrupt signal actually only reaches the thread in 3 instances. When the thread is in a wait(), join(), or sleep() method. So a thread can only be interrupted when it isn't doing anything. (In case you didn't know, a thread is in wait() while it is waiting for its chance to use the processor, and in a join() when the application is telling it to finish its last task and then terminate). So if you send an interrupt signal to a thread while it is performing some task, a flag will be marked and next time the thread is waiting, the JVM will check the flag and terminate the thread.
I must give credit to the Software Craftsmanship Articles 8-11 written by Uncle Bob, since they walked through many of the same steps I listen here.
Tuesday, February 16, 2010
Day 30
I am sorry, but I had just a very unpleasant Git rebasing experience, where there was a conflict in just about every line of every file, but of course it only showed me one or two conflicted files at a time. Thus I would remove a flurry of retarded conflicts, add the files and then rebase --continue, and once gain there would be more stupidly conflicted files. I was trying to rebase about 20 commits at once, probably not a good idea in the first place, but either way I failed to pick a good commit to rebase onto I guess, and thus my rebase failed. Now for whatever reason a few of the tests aren't passing and I don't know why.
A URL, or Uniform Resource Locator, is the standard addresser used for most http interaction; however it is not limited to http.
A URL is composed of two main parts, the protocol identifier and the resource name.
http:// www.starcraft2.com
|____| | ___________________|
protocol resource name
There are several different types of protocols, other than http, like FTP (File Transfer Protocol) or one we all see quite often, File. The File protocol is the one used locally on your computer to navigate your system.
The resource name has 4 different components, 2 of which are optional and often implied.
There is the:
Host Name - the name of the machine where the resource lives - for the starcraft2 URL this is the full www.starcraft2.com
File Name - path name to the file on the machine - if you went to www.starcraft2.com/features, the /features portion is both the file name, and what is called a 'relative URL' (meaning it is a locator relative to the base URL)
Port Number - this is typically implied, but can also be specified. For most sites you will ever go to, the port number is 80
Reference - a reference to an anchor in a specified location in a file. This is optional, and you typically wont notice this.
I will expand the resource name to its full form so you can see the components:
www.yahoo.com:80/index.html
|______________| |__| |_________| Host name - Port - File name
For most browsers, when the site ends with .com/ this implies a /index.html This isn't universally true, but for most websites that is the case. For example, this doesn't work for www.starcraft2.com because, I believe, their site is so pro and Flash based, that they find these norms irrelevant (try typing it, the error page is funny).
Connecting to a page, without a browser, is actually pretty easy. One easy way is to go to your terminal and type:
curl www.starcraft2.com this wont actually give you a connection, but will just print out the source of the page. If you want to save the source using curl, you can type:
curl www.starcraft2.com >> ~/Desktop/starcraftSource.txt
Most languages have built in methods to connect to a URL, and I know you are all gonna hate it, but I will show you the JAVA way since I am making a Java Socket Server.
This will open a connection, grab all the html, and print it out. There are other things you can do with the connection, like write to the server.
Most html pages have 'forms' - GUI objects that allow interaction with the site. User to HTML page interaction is written into the URL by your browser, and then sent to the server. Once the server receives the new URL, it process it, builds a response, and sends back a bunch more HTML for the browser to view.
Lots of HTML forms use HTML POST METHOD to send data to the server. This is called 'posting to a URL'. The server recognizes a post, and then responds.
If you know what objects are meant for interaction, or the form the URL takes after an interaction, then you can just write that URL directly to your connection, and see the response.
Later today I will go over a lot of the interesting Threading issues involved with making a Socket Server.
A URL, or Uniform Resource Locator, is the standard addresser used for most http interaction; however it is not limited to http.
A URL is composed of two main parts, the protocol identifier and the resource name.
http:// www.starcraft2.com
|____| | ___________________|
protocol resource name
There are several different types of protocols, other than http, like FTP (File Transfer Protocol) or one we all see quite often, File. The File protocol is the one used locally on your computer to navigate your system.
The resource name has 4 different components, 2 of which are optional and often implied.
There is the:
Host Name - the name of the machine where the resource lives - for the starcraft2 URL this is the full www.starcraft2.com
File Name - path name to the file on the machine - if you went to www.starcraft2.com/features, the /features portion is both the file name, and what is called a 'relative URL' (meaning it is a locator relative to the base URL)
Port Number - this is typically implied, but can also be specified. For most sites you will ever go to, the port number is 80
Reference - a reference to an anchor in a specified location in a file. This is optional, and you typically wont notice this.
I will expand the resource name to its full form so you can see the components:
www.yahoo.com:80/index.html
|______________| |__| |_________| Host name - Port - File name
For most browsers, when the site ends with .com/ this implies a /index.html This isn't universally true, but for most websites that is the case. For example, this doesn't work for www.starcraft2.com because, I believe, their site is so pro and Flash based, that they find these norms irrelevant (try typing it, the error page is funny).
Connecting to a page, without a browser, is actually pretty easy. One easy way is to go to your terminal and type:
curl www.starcraft2.com this wont actually give you a connection, but will just print out the source of the page. If you want to save the source using curl, you can type:
curl www.starcraft2.com >> ~/Desktop/starcraftSource.txt
Most languages have built in methods to connect to a URL, and I know you are all gonna hate it, but I will show you the JAVA way since I am making a Java Socket Server.
import java.net.*;
import java.io.*;
public class URLReader {
public static void main(String[] args) throws Exception {
URL sc2 = new URL("www.starcraft2.com");
URLConnection sc2connection = sc2.openConnection();
BufferedReader br = new BufferedReader(new InputStreamReader(
sc2connection.openStream()));
String inputLine;
while ((inputLine = in.readLine()) != null)
System.out.println(inputLine);
in.close();
}
}
This will open a connection, grab all the html, and print it out. There are other things you can do with the connection, like write to the server.
Most html pages have 'forms' - GUI objects that allow interaction with the site. User to HTML page interaction is written into the URL by your browser, and then sent to the server. Once the server receives the new URL, it process it, builds a response, and sends back a bunch more HTML for the browser to view.
Lots of HTML forms use HTML POST METHOD to send data to the server. This is called 'posting to a URL'. The server recognizes a post, and then responds.
If you know what objects are meant for interaction, or the form the URL takes after an interaction, then you can just write that URL directly to your connection, and see the response.
Later today I will go over a lot of the interesting Threading issues involved with making a Socket Server.
Monday, February 15, 2010
Day 29
Programming a java HTTP Socket Server is proving to be quite interesting. I spent several hours today just reading and taking notes of the Hypertext Transfer Protocols and all the complexity of moving information around the internet. Then I actually got some help from the Craftsmanship Articles written by Uncle Bob, which I had read last year. It turns out that Alphonse and Jerry created their own Socket Server in java, which is quite similar to a part of what I need to do. I also just got done reading a rather interesting article about what it means to program an AI, and it turned the idea of human intelligence on its head, if i may.
There is a plethora of detail in HTTP, and the RFC documents are very dense and a tough read. I found the only way I can really absorb much from it is by taking notes and trying to spit back the knowledge I acquired from these documents.
HTTP has gone through a few versions now, starting with HTTP/0.9 which was extremely basic, and evolving into HTTP/1.1 which seems to be the standard these days. In its most basic form, all the HTTP is, is a standard way to ask for or send to a program some packets of data.
All HTTP interactions happen through a Request - Response form. First a client (a program connected to a server) sends a request to a server. Once the server receives this request, it processes it, and builds and sends the appropriate response. If the client wants any further information, it must package up and send another request. Often there are several intermediaries as well, such as Proxies or Gateways, which will receive a message from the client and then perhaps perform some manipulation before forwarding the message to the server.
Most programs using HTTP to interact will also have a cache. This is some local memory used to store different requests and responses, already packaged and ready to be sent. Using a cache can save a server or client a lot of processing time, so that the only real limitation they face will be the time it takes to get back a response.
Caches can also be particularly useful if there several proxies, gateways, or intermediate servers sitting between the Client and the Origin Server. If, for example, a client needs to send a request through proxy A and B before reaching the server, then both proxy A and proxy B can build up a cache to save time. Proxy A could save the Client's request, and the response coming back from the Origin Server. This way, the next time Proxy A gets that same request, it can just send the stored response right away without having to wait to hear back from the server.
When I say 'package' a request or response, I mean forming a proper HTTP message. The desired data in any message is called the entity of that message. The entity can't be sent on its own because it might not be in a familiar form for many different programs on the web, so it must be packaged into a formated HTTP message. These typically consist of a few header-fields which define certain basics about the entity, like its length or perhaps encryption type, and then the message-body which will contain the entity.
Typically if an HTTP formated message wants to send complicated data structures it will use MIME types. A MIME, Multi-purpose Internet Mail Extension, is just a standard way or form to send complicated data in HTTP messages. There are a variety of MIMEs, and I have much more to learn about them in order to understand how they actually vary.
A Socket Server is a type of server that uses Sockets to handle multiple requests. Say you have a server hosting your website which provides some service to clients. The most basic of servers will be able to talk to just one client at a time, and will focus all of its attention at that one client (spending far far too much time just waiting for the next request); but say you want your website talking to multiple clients at the same time. You might use a Socket Server to create a Server Socket for each port you wish to talk on, and then generate a Socket for each new client it wishes to talk to. So a Socket Server will make Server Sockets. A Server Socket is a place holder that is watching for any communication on a port, like port 80 (the standard port for communication on the internet). Once the Server Socket sees a client trying to talk to your server on its specified port, it will create a Socket for the client to talk to. The Socket will then handle any requests from that one client, and send responses back once it is able to get some processor time to use the program sitting on your website.
Its been pretty cool learning how to get programs talking over the web, and I am excited to make my own socket server and see it talk with my browser!
There is a plethora of detail in HTTP, and the RFC documents are very dense and a tough read. I found the only way I can really absorb much from it is by taking notes and trying to spit back the knowledge I acquired from these documents.
HTTP has gone through a few versions now, starting with HTTP/0.9 which was extremely basic, and evolving into HTTP/1.1 which seems to be the standard these days. In its most basic form, all the HTTP is, is a standard way to ask for or send to a program some packets of data.
All HTTP interactions happen through a Request - Response form. First a client (a program connected to a server) sends a request to a server. Once the server receives this request, it processes it, and builds and sends the appropriate response. If the client wants any further information, it must package up and send another request. Often there are several intermediaries as well, such as Proxies or Gateways, which will receive a message from the client and then perhaps perform some manipulation before forwarding the message to the server.
Most programs using HTTP to interact will also have a cache. This is some local memory used to store different requests and responses, already packaged and ready to be sent. Using a cache can save a server or client a lot of processing time, so that the only real limitation they face will be the time it takes to get back a response.
Caches can also be particularly useful if there several proxies, gateways, or intermediate servers sitting between the Client and the Origin Server. If, for example, a client needs to send a request through proxy A and B before reaching the server, then both proxy A and proxy B can build up a cache to save time. Proxy A could save the Client's request, and the response coming back from the Origin Server. This way, the next time Proxy A gets that same request, it can just send the stored response right away without having to wait to hear back from the server.
When I say 'package' a request or response, I mean forming a proper HTTP message. The desired data in any message is called the entity of that message. The entity can't be sent on its own because it might not be in a familiar form for many different programs on the web, so it must be packaged into a formated HTTP message. These typically consist of a few header-fields which define certain basics about the entity, like its length or perhaps encryption type, and then the message-body which will contain the entity.
Typically if an HTTP formated message wants to send complicated data structures it will use MIME types. A MIME, Multi-purpose Internet Mail Extension, is just a standard way or form to send complicated data in HTTP messages. There are a variety of MIMEs, and I have much more to learn about them in order to understand how they actually vary.
A Socket Server is a type of server that uses Sockets to handle multiple requests. Say you have a server hosting your website which provides some service to clients. The most basic of servers will be able to talk to just one client at a time, and will focus all of its attention at that one client (spending far far too much time just waiting for the next request); but say you want your website talking to multiple clients at the same time. You might use a Socket Server to create a Server Socket for each port you wish to talk on, and then generate a Socket for each new client it wishes to talk to. So a Socket Server will make Server Sockets. A Server Socket is a place holder that is watching for any communication on a port, like port 80 (the standard port for communication on the internet). Once the Server Socket sees a client trying to talk to your server on its specified port, it will create a Socket for the client to talk to. The Socket will then handle any requests from that one client, and send responses back once it is able to get some processor time to use the program sitting on your website.
Its been pretty cool learning how to get programs talking over the web, and I am excited to make my own socket server and see it talk with my browser!
Friday, February 12, 2010
Day 28
So I first must apologize for not blogging on Thursday. I know day 26 says Thursday, but that was actually done Wednesday night, and then I fell asleep before I submitted, so I submitted it right when I woke up. Thursday I was still trapped in Philadelphia, and Micah spent the whole morning trying to find a flight home. I was feeling under weather, and we were riding a roller coaster, trying time and time again to get on a flight. Finally after a few attempts, we made it to the airport for a flight that had not yet been canceled. We then had a very anti climatic sprint to get to our gate, only to find out that we had to wait another hour and a half for the airline to find another flight attendant. I didn't get home until around 6:00 and we had lost an hour from the flight, so the day was quite unproductive, other than the fact that we made it home.
Today, I had hoped to make up for the lack of work completed on Thursday, unfortunately today was manual labor day. There were several tasks that needed to be done at the office, and being the apprentice - I was put to work. I literally got about 5 lines of code down today. On a more lightening note, one side of the office looks radically different now!
First I had to assemble 4 new chairs, which was actually a fairly painless process.
Then came a broken toilet that needed fixing. A relatively quick trip to ACE and a little tweaking to get the flush mechanism operating at its full potential. And let me tell you, there are under par flush handles and there are excellent flush handles. My first go at it, Micah was disappointed at the give of the handle and that it required so much motion before the flushing actually began. On my second go at it, I successfully measured the exact give to action ratio to create a handle that is both satisfyingly smooth in its motion as well as efficient in its reaction to the toilet's depositor.
After the toilet came the real task. This was actually after I had ordered lunch for everyone, though I might add that I failed in this endeavor because I forgot we had a vegetarian in the office. Fortunately, the pizza place had failed to deliver on my failed request, and thus two wrong made a right and there was no meat on the pizza. Unfortunately, this was discovered after Colin (he who prefers the veggies over the remains of a carcass) had already compensated for my thoughtlessness and created a delightful Peanut Butter, Pretzel, Banana, and Jelly sandwich on Rye bread.
So my real task came after this lunch, just as I thought I might be able to get some coding done, when some guys brought in 4 brand new tables that needed assembly. Not just the IKEA - stick some pre-made pegs in pre-drilled holes. No, these were big, heavy, and highly demanding tables which give you no assistance on assembly. As if tables can give you assistance... but either way, these were beastly projects. Each had four legs with a base that had 8 screws which had to be positioned after first measuring the proper distances from each side. Before the screws could be placed, I had to drill little holes to guide the screws in properly. So in total, each table required about 12 measurements, 16 pencil markings, 32 holes drilled, 32 screws screwed in the guided holes, and then finally 4 legs to be screwed onto their respective base paltes. I tried making a measurement template after the first table (as Doug suggested), but we had no paper large enough to span the table and the cardboard template I tried was too small and too inaccurate.
What made it all far... far worse, was the electric drill ran out of battery before even the first table was completed. As a result, the majority of the screws were placed in by hand through a variety of combinations of unintended uses for tools. If you never have screwed in thick screws into fairly hardwood... it isn't an easy task. I am a fairly strong fellow, and even using my full body weight, sometimes I couldn't get the screws to budge. Thank goodness for mechanical advantage!
I am happy to say, that even though I was sweating at the job for a very long time, I defeated Doug's expectations that I would still be there when he returned on Monday! Take that Mr. IOnlyUsePoweredDrills ( go java CamelCase) !! No offense intended of course, its actually a reflection of frustration for 8th Light owning a battery powered drill with no backup battery.
This weekend I will makeup for all the lost time these last two days, and then start on my new assignment of creating an HTTP Server from scratch.
Today, I had hoped to make up for the lack of work completed on Thursday, unfortunately today was manual labor day. There were several tasks that needed to be done at the office, and being the apprentice - I was put to work. I literally got about 5 lines of code down today. On a more lightening note, one side of the office looks radically different now!
First I had to assemble 4 new chairs, which was actually a fairly painless process.
Then came a broken toilet that needed fixing. A relatively quick trip to ACE and a little tweaking to get the flush mechanism operating at its full potential. And let me tell you, there are under par flush handles and there are excellent flush handles. My first go at it, Micah was disappointed at the give of the handle and that it required so much motion before the flushing actually began. On my second go at it, I successfully measured the exact give to action ratio to create a handle that is both satisfyingly smooth in its motion as well as efficient in its reaction to the toilet's depositor.
After the toilet came the real task. This was actually after I had ordered lunch for everyone, though I might add that I failed in this endeavor because I forgot we had a vegetarian in the office. Fortunately, the pizza place had failed to deliver on my failed request, and thus two wrong made a right and there was no meat on the pizza. Unfortunately, this was discovered after Colin (he who prefers the veggies over the remains of a carcass) had already compensated for my thoughtlessness and created a delightful Peanut Butter, Pretzel, Banana, and Jelly sandwich on Rye bread.
So my real task came after this lunch, just as I thought I might be able to get some coding done, when some guys brought in 4 brand new tables that needed assembly. Not just the IKEA - stick some pre-made pegs in pre-drilled holes. No, these were big, heavy, and highly demanding tables which give you no assistance on assembly. As if tables can give you assistance... but either way, these were beastly projects. Each had four legs with a base that had 8 screws which had to be positioned after first measuring the proper distances from each side. Before the screws could be placed, I had to drill little holes to guide the screws in properly. So in total, each table required about 12 measurements, 16 pencil markings, 32 holes drilled, 32 screws screwed in the guided holes, and then finally 4 legs to be screwed onto their respective base paltes. I tried making a measurement template after the first table (as Doug suggested), but we had no paper large enough to span the table and the cardboard template I tried was too small and too inaccurate.
What made it all far... far worse, was the electric drill ran out of battery before even the first table was completed. As a result, the majority of the screws were placed in by hand through a variety of combinations of unintended uses for tools. If you never have screwed in thick screws into fairly hardwood... it isn't an easy task. I am a fairly strong fellow, and even using my full body weight, sometimes I couldn't get the screws to budge. Thank goodness for mechanical advantage!
I am happy to say, that even though I was sweating at the job for a very long time, I defeated Doug's expectations that I would still be there when he returned on Monday! Take that Mr. IOnlyUsePoweredDrills ( go java CamelCase) !! No offense intended of course, its actually a reflection of frustration for 8th Light owning a battery powered drill with no backup battery.
This weekend I will makeup for all the lost time these last two days, and then start on my new assignment of creating an HTTP Server from scratch.
Thursday, February 11, 2010
Day 26
"Do not try to bend the spoon. That's impossible. Instead... only try to realize the truth. There is no spoon. Then you'll see, that it is not the spoon that bends, it is only yourself."
I wanted to place that quote at the end, but it seemed like such a fitting introduction that it belonged at the start.
Making massive changes to your code, without changing a single line.
Often you will find yourself working on a project, when an unexpected requirement pops up and seems to demand that you perform a complete redesign or restructuring of your code. You might then find you have no idea where to start or how to proceed. You look around for awhile, testing out small changes in various places, but never getting the feel that you are going in the right direction. Finally, you stumble across an area that looks important, so you make a change that seems logical, watch as your tests fail in the way you expect, and then proceed to change your tests to fit a new criterion. However, once you get to your tests you realize that they are actually testing exactly what you had wanted them to! These tests, which are at the center of your massive overhaul, are properly testing both your old system and your new system... but you know that this is supposed to be a big change! What's going on?
This exact thing happened to me the other day. I had developed a system for weeks, carefully designing each facet, and it was almost just the way I wanted it. Then I suddenly discovered there was a fundamental change that was needed to fulfill a requirement. Not knowing where to start, I began to fiddle around in a variety of places trying to get a foothold on what I needed to do. Finally, I made a change to the code, saw a test fail, then changed the test... but the change didn't feel right. I made the change anyway, and observed the green flash before me once again. It was at that point that I realized this was all wrong. I realized that I wasn't trying to change the code at all, I was just trying to change what it meant. The massive overhaul and fundamental change that needed to occur was in my mind.
After the fact, it seemed rather obvious, but I got trapped into thinking- since my code only did one thing, it could only mean one thing. This, of course, was not the case. I was making a change that would affect every part of my system, but would change none of the functionality, just the interpretation.
I found a few signals that indicate when the change is in the theme and not in the scheme.
The biggest is that you have no idea where to make a change. You designed this system from the start and you know it inside and out, yet you can't seem to find what you need to tweak to get started on this big change.
Another is that the tests are yelling at you for trying to change them. You seem to think you know what to change, but when you make the according change to the tests it feels completely wrong. The tests seem to fight the change.
The last big one I can think of is you can't completely understand what this change means in terms of your code. You know what it should allow you to do, but you don't quite understand the new mind set you are supposed to settle into.
These three things likely indicate that either your system is already the way it is meant to be and you just need to figure out why, or that your system sucks and you should rewrite it all. It could also mean that your original problem is far more complicated that you first anticipated, but if thats the case your tests are probably screaming at you because they unsatisfied or just plain wrong.
Basically, try to remember to 'Free your mind'
I wanted to place that quote at the end, but it seemed like such a fitting introduction that it belonged at the start.
Making massive changes to your code, without changing a single line.
Often you will find yourself working on a project, when an unexpected requirement pops up and seems to demand that you perform a complete redesign or restructuring of your code. You might then find you have no idea where to start or how to proceed. You look around for awhile, testing out small changes in various places, but never getting the feel that you are going in the right direction. Finally, you stumble across an area that looks important, so you make a change that seems logical, watch as your tests fail in the way you expect, and then proceed to change your tests to fit a new criterion. However, once you get to your tests you realize that they are actually testing exactly what you had wanted them to! These tests, which are at the center of your massive overhaul, are properly testing both your old system and your new system... but you know that this is supposed to be a big change! What's going on?
This exact thing happened to me the other day. I had developed a system for weeks, carefully designing each facet, and it was almost just the way I wanted it. Then I suddenly discovered there was a fundamental change that was needed to fulfill a requirement. Not knowing where to start, I began to fiddle around in a variety of places trying to get a foothold on what I needed to do. Finally, I made a change to the code, saw a test fail, then changed the test... but the change didn't feel right. I made the change anyway, and observed the green flash before me once again. It was at that point that I realized this was all wrong. I realized that I wasn't trying to change the code at all, I was just trying to change what it meant. The massive overhaul and fundamental change that needed to occur was in my mind.
After the fact, it seemed rather obvious, but I got trapped into thinking- since my code only did one thing, it could only mean one thing. This, of course, was not the case. I was making a change that would affect every part of my system, but would change none of the functionality, just the interpretation.
I found a few signals that indicate when the change is in the theme and not in the scheme.
The biggest is that you have no idea where to make a change. You designed this system from the start and you know it inside and out, yet you can't seem to find what you need to tweak to get started on this big change.
Another is that the tests are yelling at you for trying to change them. You seem to think you know what to change, but when you make the according change to the tests it feels completely wrong. The tests seem to fight the change.
The last big one I can think of is you can't completely understand what this change means in terms of your code. You know what it should allow you to do, but you don't quite understand the new mind set you are supposed to settle into.
These three things likely indicate that either your system is already the way it is meant to be and you just need to figure out why, or that your system sucks and you should rewrite it all. It could also mean that your original problem is far more complicated that you first anticipated, but if thats the case your tests are probably screaming at you because they unsatisfied or just plain wrong.
Basically, try to remember to 'Free your mind'
Tuesday, February 9, 2010
Day 25
It appears I will be trapped in Philadelphia until Friday because of severe weather conditions! It is supposed to snow 10-15 inches tonight, so tonight we all went and stocked up on groceries and supplies.
Anyway, today I was got to sit in on a very engaging and enlightening meeting with a new client of 8th Light's. Micah, Paul, Craig, and I were introduced to this companies exec board, who then sat down with us and described their dilemma. They had a huge amount of data, but they had to use other companies to access it. Some of the data was collected through one of their websites, which was created and hosted by a company they hired, and as such they had to go through this company to get any of their data. There is also a huge data bank which they do not own, but have a license to use; yet they still had been going through a third party to gain access to this data.
The gist of it is that they wanted all this data under one roof. Our client knew the power of this knowledge, and they wanted to hold it and completely own it. So 8th Light's job would be to create a system that could store and interact with this data, after merging a huge amount of data in from several sources.
From an apprentice stand point, this was a gold mine to observe. I got to meet some very ambitious and driven people, who couldn't help but think big. I got to see the manner in which a Craftsman might respond to the requests and dreams of enthusiastic clients, and it was always in the affirmative. And I also got to see the outlining of a project from nothing to something very achievable.
First the client described their debacle, and then we would ask chains of questions to get a deeper understanding so that we could formulate possible solutions. Then the client explained what they were looking for, and what they imagined their system would do, and once again we asked questions to contemplate how we might implement this system. Next, Micah and Paul began outlining what it meant to use the Agile process, or to work in short iterations with rigorous customer interaction and frequent releases. We proceeded to define several Epics, or big chunks of functionality that the client thought were crucial. Once these Epic had been ordered, or at least once we knew which one had to be first, we then broke the Epic into Stories and estimated. Finally, we showed them the stories with the estimates, and gave them an idea of how soon we could get them the functionality they desired.
All in all a very educational experience. I now have an understanding of how to began a project from scratch, and how to lead the customer along your chain of thought. I am also remarkably exhausted from waking up extremely early and getting a mere hour of sleep. must collapse... arrrgggghhh... ZZzzzz
Anyway, today I was got to sit in on a very engaging and enlightening meeting with a new client of 8th Light's. Micah, Paul, Craig, and I were introduced to this companies exec board, who then sat down with us and described their dilemma. They had a huge amount of data, but they had to use other companies to access it. Some of the data was collected through one of their websites, which was created and hosted by a company they hired, and as such they had to go through this company to get any of their data. There is also a huge data bank which they do not own, but have a license to use; yet they still had been going through a third party to gain access to this data.
The gist of it is that they wanted all this data under one roof. Our client knew the power of this knowledge, and they wanted to hold it and completely own it. So 8th Light's job would be to create a system that could store and interact with this data, after merging a huge amount of data in from several sources.
From an apprentice stand point, this was a gold mine to observe. I got to meet some very ambitious and driven people, who couldn't help but think big. I got to see the manner in which a Craftsman might respond to the requests and dreams of enthusiastic clients, and it was always in the affirmative. And I also got to see the outlining of a project from nothing to something very achievable.
First the client described their debacle, and then we would ask chains of questions to get a deeper understanding so that we could formulate possible solutions. Then the client explained what they were looking for, and what they imagined their system would do, and once again we asked questions to contemplate how we might implement this system. Next, Micah and Paul began outlining what it meant to use the Agile process, or to work in short iterations with rigorous customer interaction and frequent releases. We proceeded to define several Epics, or big chunks of functionality that the client thought were crucial. Once these Epic had been ordered, or at least once we knew which one had to be first, we then broke the Epic into Stories and estimated. Finally, we showed them the stories with the estimates, and gave them an idea of how soon we could get them the functionality they desired.
All in all a very educational experience. I now have an understanding of how to began a project from scratch, and how to lead the customer along your chain of thought. I am also remarkably exhausted from waking up extremely early and getting a mere hour of sleep. must collapse... arrrgggghhh... ZZzzzz
Monday, February 8, 2010
Day 24
So while driving home from work today, I drove by a house which left their garage door open. They had left their car out in the snow because their garage was so overwhelmingly packed with stuff that they no longer had room for their car. When I saw this, the two things came to mind. The first was my Grandma, who is the type of person who could never throw anything away, and as a result has no room in her garage. The second was what a program would look like if she had written it.
Often I, and many other developers I know, have the temptation to comment out some code and keep it around just in case it might come in handy later. This happens quite often when you are spiking, or when (and god knows why) you are not using version control. You write a tid-bit of code, find a better way to rewrite the code, but save the old code just in case your new idea fails. Do this over and over, and things can get pretty ugly.
I recently wrote some C and X86 assembly to write an operating system for one of my classes. My group and I really had no idea how to start going about writing our OS, so quite frequently we would save old code we had written so that we wouldn't have to reinvent the wheel. As a result, in some files we had to doing some hunting just to find the real production code through the haze of the comments. This is a Dirty Garage.
Uncle Bob uses the metaphor of a Dirty Kitchen for poorly written code. If you are a chef working at a restaurant on a very busy night, and rather than cleaning up pans or knives after every use, you instead just toss the dirty ones aside and grab the nearest clean on. At first, you will be able to get a bunch of dinners out quite rapidly, since you waste no time cleaning anything; however, as the night progresses you will find it more and more difficult to get any clean pans and eventually you virtually stop cooking all together and spend most of your time hunting down clean tools. The same thing is true with software. If you rush to get a lot done really quickly, and you push off keeping your code clean, then eventually you will hit a point where you are spending more time trying to figure out what is going on instead of developing.
If my Grandma were to write a program, I imagine that it would be so densely packed with commented and saved code, that unveiling any functionality would be quite a challenge. This is a Dirty Garage. Where you have so much saved junk in your files that you no longer have any room for that which is meant to occupy your production files, your functioning code. Where you fear deleting code because you think it might be useful again in the future. Sometimes this can be true, and if you are just messing around and trying to refactor, often commenting out code for later or for reference can be valuable. Often though, you will comment out a block, forget about it, and it will linger and rot your code. Perhaps you hand your project off to another developer who is too afraid to delete the commented code because he/she thinks it might have some unspoken value.
Although a Dirty Kitchen can be more harmful than a Dirty Garage... no one wants to park their car in a garage full of junk.
Often I, and many other developers I know, have the temptation to comment out some code and keep it around just in case it might come in handy later. This happens quite often when you are spiking, or when (and god knows why) you are not using version control. You write a tid-bit of code, find a better way to rewrite the code, but save the old code just in case your new idea fails. Do this over and over, and things can get pretty ugly.
I recently wrote some C and X86 assembly to write an operating system for one of my classes. My group and I really had no idea how to start going about writing our OS, so quite frequently we would save old code we had written so that we wouldn't have to reinvent the wheel. As a result, in some files we had to doing some hunting just to find the real production code through the haze of the comments. This is a Dirty Garage.
Uncle Bob uses the metaphor of a Dirty Kitchen for poorly written code. If you are a chef working at a restaurant on a very busy night, and rather than cleaning up pans or knives after every use, you instead just toss the dirty ones aside and grab the nearest clean on. At first, you will be able to get a bunch of dinners out quite rapidly, since you waste no time cleaning anything; however, as the night progresses you will find it more and more difficult to get any clean pans and eventually you virtually stop cooking all together and spend most of your time hunting down clean tools. The same thing is true with software. If you rush to get a lot done really quickly, and you push off keeping your code clean, then eventually you will hit a point where you are spending more time trying to figure out what is going on instead of developing.
If my Grandma were to write a program, I imagine that it would be so densely packed with commented and saved code, that unveiling any functionality would be quite a challenge. This is a Dirty Garage. Where you have so much saved junk in your files that you no longer have any room for that which is meant to occupy your production files, your functioning code. Where you fear deleting code because you think it might be useful again in the future. Sometimes this can be true, and if you are just messing around and trying to refactor, often commenting out code for later or for reference can be valuable. Often though, you will comment out a block, forget about it, and it will linger and rot your code. Perhaps you hand your project off to another developer who is too afraid to delete the commented code because he/she thinks it might have some unspoken value.
Although a Dirty Kitchen can be more harmful than a Dirty Garage... no one wants to park their car in a garage full of junk.
Sunday, February 7, 2010
Day 23
A light weight and 'humble' user interface is the way to go. When making a UI for some application of website it is important to make sure you draw a line between your logic and your interface. Typically you don't want anything important happening in a text box or a button. You want the button to only know where it is, when its clicked, and how it looks.
Say your making a calculator application. This calculator has a screen, to display results, and a bunch of buttons for numbers and operations.
First lets look at what happens if you program this the bad way.
You start making the application by first making a main class to store which buttons have been pressed and all the needed info, and then by creating a bunch of buttons. You write a test to make sure there is a number '1' button, and then you make this button. Then you test that when pressed, this button will output a 1. You proceed to do make a new button for all the number 0-9, each button being specifically designed for its number. Every single one of these buttons have a set of tests that are virtually identical, checking to make sure this button is a certain size, making sure it is clickable, making sure it changes form when being clicked, making sure it tells your main class to record the button's info. All of these tests for every button, and then you add 1 different tests that makes sure then number 1 button outputs a 1, and a 2 outputs a 2, and so forth.
Already you have huge amounts of duplication and a rigid form, but it gets worse. Next you start making the operation buttons. You make an add button, once again repeating all the steps for a button but this time adding a few extra tests to make sure this button takes inputs from the main class. Then you make a multiply button (and here is where it gets bad), but you realize your gonna just have to do some addition several times, and you don't want to rewrite the addition code... So you tell your multiply button to 'press' the add button over and over to get multiplication. This is sick. Buttons should not be pressing other buttons! But you persevere. You proceed to make subtraction and division in the same manner with the button's 'clicked()' function doing all the math.
Finally you put all these buttons on a panel and have main display the inputs and outputs on some text panel and show your friends your cool new calculator. Then one of them asks "Hey this is really swell, but can you make a version for the terminal? I want to be able to type 1 + 1 = ? and get a result,". You of course, being the savvy and eager coder that you are, say "Most certainly! I already have all the logic, I just have to change the inputs!". So you go back to your program and start trying to figure out how you will change the inputs. As you scour for ways, you realize even if you changed the inputs all of your operations are inside the buttons! You can't just erase all your code and you know you shouldn't copy paste it somewhere else because then you loose all your tests. You can't bring your tests because they all test the operations by pretending to click the buttons. And you realize you are stuck...
Then you commit the sin of all sins, and make your terminal based calculator by getting the text inputs and then telling your terminal app to... press buttons...
Not only does tying logic to the UI force you to repeat a ton of code that can't be reused, it also makes for rigid tests that are dependent on your UI.
When you do it the correct way, you make a very light UI and keep all your logic behind closed doors. You start by making your main calculator class. Then you add in all of your operations, no buttons yet, making sure that they don't care who is using them, they just take numbers in and spit numbers out. Once you have all your operations done and tested, you design a single button. This button is simple, it just knows about buttony things like where it is and when its pressed.
Next you make a view class, which has a bunch of instances of these buttons, but all with different names like '1' and 'ADD', and yet all they do is get pressed and tell the view they were pressed. The view, or this humble dialog, will keep track of all the basic information. When a button is pressed, the view records it and saves the needed data. If an operation button is pressed, the view makes sure it has the needed data, and then sends it all to the main calculator class and tell it do perform the operation. The main class then sends the result right back to the view, which promptly displays it.
Using this sort of design, you can keep all of your business logic well isolated and tested, and you get to make any sort of UI you want. Say your friend sees your new calculator and asks you to make the terminal version now. This is easy as pie, all you have to do is make another humble view!
Say your making a calculator application. This calculator has a screen, to display results, and a bunch of buttons for numbers and operations.
First lets look at what happens if you program this the bad way.
You start making the application by first making a main class to store which buttons have been pressed and all the needed info, and then by creating a bunch of buttons. You write a test to make sure there is a number '1' button, and then you make this button. Then you test that when pressed, this button will output a 1. You proceed to do make a new button for all the number 0-9, each button being specifically designed for its number. Every single one of these buttons have a set of tests that are virtually identical, checking to make sure this button is a certain size, making sure it is clickable, making sure it changes form when being clicked, making sure it tells your main class to record the button's info. All of these tests for every button, and then you add 1 different tests that makes sure then number 1 button outputs a 1, and a 2 outputs a 2, and so forth.
Already you have huge amounts of duplication and a rigid form, but it gets worse. Next you start making the operation buttons. You make an add button, once again repeating all the steps for a button but this time adding a few extra tests to make sure this button takes inputs from the main class. Then you make a multiply button (and here is where it gets bad), but you realize your gonna just have to do some addition several times, and you don't want to rewrite the addition code... So you tell your multiply button to 'press' the add button over and over to get multiplication. This is sick. Buttons should not be pressing other buttons! But you persevere. You proceed to make subtraction and division in the same manner with the button's 'clicked()' function doing all the math.
Finally you put all these buttons on a panel and have main display the inputs and outputs on some text panel and show your friends your cool new calculator. Then one of them asks "Hey this is really swell, but can you make a version for the terminal? I want to be able to type 1 + 1 = ? and get a result,". You of course, being the savvy and eager coder that you are, say "Most certainly! I already have all the logic, I just have to change the inputs!". So you go back to your program and start trying to figure out how you will change the inputs. As you scour for ways, you realize even if you changed the inputs all of your operations are inside the buttons! You can't just erase all your code and you know you shouldn't copy paste it somewhere else because then you loose all your tests. You can't bring your tests because they all test the operations by pretending to click the buttons. And you realize you are stuck...
Then you commit the sin of all sins, and make your terminal based calculator by getting the text inputs and then telling your terminal app to... press buttons...
Not only does tying logic to the UI force you to repeat a ton of code that can't be reused, it also makes for rigid tests that are dependent on your UI.
When you do it the correct way, you make a very light UI and keep all your logic behind closed doors. You start by making your main calculator class. Then you add in all of your operations, no buttons yet, making sure that they don't care who is using them, they just take numbers in and spit numbers out. Once you have all your operations done and tested, you design a single button. This button is simple, it just knows about buttony things like where it is and when its pressed.
Next you make a view class, which has a bunch of instances of these buttons, but all with different names like '1' and 'ADD', and yet all they do is get pressed and tell the view they were pressed. The view, or this humble dialog, will keep track of all the basic information. When a button is pressed, the view records it and saves the needed data. If an operation button is pressed, the view makes sure it has the needed data, and then sends it all to the main calculator class and tell it do perform the operation. The main class then sends the result right back to the view, which promptly displays it.
Using this sort of design, you can keep all of your business logic well isolated and tested, and you get to make any sort of UI you want. Say your friend sees your new calculator and asks you to make the terminal version now. This is easy as pie, all you have to do is make another humble view!
Thursday, February 4, 2010
Day 22
I need to make today's post very brief because I have been very busy today, and really want to finish my current project so I can get started on something new tomorrow.
I wish I had something really cool or interesting to say, but today I spent a very large amount of my time doing very tedious and rather annoying work. Making User Interfaces is pretty boring. Well... I should be more clear. The logic behind the UI can be a lot of fun. I have learned a lot about the complexities and everything that happens behind the scenes with User Inputs and such, but the actual scene is realllyyy annoying to make.
I spent hours today just trying to get a cursor (or as a later learned, it is actually called the caret) to pixel perfect fit the text across different types of input panels. The problem is that the text lines up in drastically different ways in, for example, a text box vs. a text area. Text boxes always have the text centered, and thus the caret must also be centered. In a text area, the text will start at the very top, and will of course span across several lines, thus the caret must operate in a fundamentally different way than in a text box.
Now if you are just trying to get everything to work, you can hack in easy ways to make the caret look right, without it actually being able to truly work right. For example you could easily code in an exact length for the caret, and exact positions for it to start on every line, but then you could never change the size of the font or the area itself. You could also design drastically different carets for a text box and a text area, but the problem is that there is just enough similarities between how they have to be made that you would end up with a lot of duplicate code.
In the end I found ways to get pixel perfect carets for the different input panels, using mostly all the same code with very little duplication, but it took large amounts of experimentation and fiddling with different offsets and such.
What made it worse was, and I hate to say this, trying to accomplish this with TDD. It was a mistake. Any change I had to make to the size of the caret, or the position/coordinates I wanted it to be at, or the manner in which it was positioned... I had to make these changes in every test as well as in the production code. This is wrong, and is not how coding should be done. I am inclined to say that when designing UIs, it is better to just put some tests in after you know you have everything looking the way you want. I mean... tests just can't tell you how something will look. You can write a test that seems to make sense in how the positioning will work, but then you run the program and everything is completely misaligned or the cursor is actually 2 pixels too long at the top and 3 pixels to short on the bottom. There are just too many nit picky little things involved in the presentation of UIs that TDD can't help you with.
Please, if you disagree or know a better way, tell me.
Friday -
Afternote:
Paul suggested I read the Humble Dialog Box by Michael Feathers, which is an excellent article on how to separate your UI from all your logic. I will blog further about this today.
I wish I had something really cool or interesting to say, but today I spent a very large amount of my time doing very tedious and rather annoying work. Making User Interfaces is pretty boring. Well... I should be more clear. The logic behind the UI can be a lot of fun. I have learned a lot about the complexities and everything that happens behind the scenes with User Inputs and such, but the actual scene is realllyyy annoying to make.
I spent hours today just trying to get a cursor (or as a later learned, it is actually called the caret) to pixel perfect fit the text across different types of input panels. The problem is that the text lines up in drastically different ways in, for example, a text box vs. a text area. Text boxes always have the text centered, and thus the caret must also be centered. In a text area, the text will start at the very top, and will of course span across several lines, thus the caret must operate in a fundamentally different way than in a text box.
Now if you are just trying to get everything to work, you can hack in easy ways to make the caret look right, without it actually being able to truly work right. For example you could easily code in an exact length for the caret, and exact positions for it to start on every line, but then you could never change the size of the font or the area itself. You could also design drastically different carets for a text box and a text area, but the problem is that there is just enough similarities between how they have to be made that you would end up with a lot of duplicate code.
In the end I found ways to get pixel perfect carets for the different input panels, using mostly all the same code with very little duplication, but it took large amounts of experimentation and fiddling with different offsets and such.
What made it worse was, and I hate to say this, trying to accomplish this with TDD. It was a mistake. Any change I had to make to the size of the caret, or the position/coordinates I wanted it to be at, or the manner in which it was positioned... I had to make these changes in every test as well as in the production code. This is wrong, and is not how coding should be done. I am inclined to say that when designing UIs, it is better to just put some tests in after you know you have everything looking the way you want. I mean... tests just can't tell you how something will look. You can write a test that seems to make sense in how the positioning will work, but then you run the program and everything is completely misaligned or the cursor is actually 2 pixels too long at the top and 3 pixels to short on the bottom. There are just too many nit picky little things involved in the presentation of UIs that TDD can't help you with.
Please, if you disagree or know a better way, tell me.
Friday -
Afternote:
Paul suggested I read the Humble Dialog Box by Michael Feathers, which is an excellent article on how to separate your UI from all your logic. I will blog further about this today.
Wednesday, February 3, 2010
Day 21
One issue I am consistently having is what I will refer to as coding A.D.D. It is drastically slowing my speed, and typically doesn't lead to anything valuable. Coding A.D.D. is where you are working on a project, either writing new production code or trying to change existing code to fit an unanticipated change, and you aren't exactly sure where to go or what to do. Instead of taking one idea through to the end and trying to at least getting something to work in a linear fashion, you work spastically and jump from idea to idea without actually following through with any of them.
To look at it from another angle, linear thinking would be throwing points onto a graph one after another all going in the same direction, and at the end you connect all the points to form an functional straight line. Coding A.D.D. is more like plotting points like throwing darts at a dart board. They are scattered and sparse, have no real pattern, and when you try to connect them all you get is a big mess.
This is what I frequently end up doing, but rather than trying to connect all the dots, I just erase all the extra code I wrote and start over again. My brain doesn't work in a linear manner, instead I tend to pop out idea after idea, each seeming better than before but really not getting me anywhere. It certainly gives me a greater understanding of the problem, when I try every solution I can think of, but I usually end up using my first strategy to get the job done anyway. It is extremely time consuming and frustrating, and worst of all, it leads to far less progress.
I find that I typically don't find the better solution to a problem until I come back to it a few hours or a few days later. When I look at it again later, the better solution pours into my mind in its entirety, and I know almost exactly how I want to improve my code. This is much more effective and efficient.
I suppose the problem is discipline. Micah said, and it seems rather obvious now, that is almost always better just to get it working first. Get some good tests, get the functionality out of the way - even if you are certain there is a better way to do it - just go with what works. Then when you come back to it, you know the worst you can do is leave it working. I also believe it is easier to refactor and redesign when you acquire extra understanding from seeing the code work as it should. This will make it easier to see a clear solution.
To look at it from another angle, linear thinking would be throwing points onto a graph one after another all going in the same direction, and at the end you connect all the points to form an functional straight line. Coding A.D.D. is more like plotting points like throwing darts at a dart board. They are scattered and sparse, have no real pattern, and when you try to connect them all you get is a big mess.
This is what I frequently end up doing, but rather than trying to connect all the dots, I just erase all the extra code I wrote and start over again. My brain doesn't work in a linear manner, instead I tend to pop out idea after idea, each seeming better than before but really not getting me anywhere. It certainly gives me a greater understanding of the problem, when I try every solution I can think of, but I usually end up using my first strategy to get the job done anyway. It is extremely time consuming and frustrating, and worst of all, it leads to far less progress.
I find that I typically don't find the better solution to a problem until I come back to it a few hours or a few days later. When I look at it again later, the better solution pours into my mind in its entirety, and I know almost exactly how I want to improve my code. This is much more effective and efficient.
I suppose the problem is discipline. Micah said, and it seems rather obvious now, that is almost always better just to get it working first. Get some good tests, get the functionality out of the way - even if you are certain there is a better way to do it - just go with what works. Then when you come back to it, you know the worst you can do is leave it working. I also believe it is easier to refactor and redesign when you acquire extra understanding from seeing the code work as it should. This will make it easier to see a clear solution.
Tuesday, February 2, 2010
Day 20
One thing I wanted to write a bit about is selling your skills to clients. Though not the immediate concern of most developers, it is still something that someone has to do. On Friday I was with Micah M. and Micah B. as we sat down to meet with a possible future client. These were the same guys who had the PHP code from the seventh realm of hell. These poor guys had an excellent idea, and they needed someone who could make it a reality.
They had a website with thousands of hits every week and nearly as many bug reports. The founder spoke about how he wished he could wake up in the morning and not dread all the emails he was going to have read about the many bugs people kept finding. They had gone through many developers and a few different companies, just trying to get any sort of website out there to start getting a fan base. After first having a group of developers fail, they found another group which got the website started, and then another group after that to try and maintain and slowly start getting rid of bugs.
These guys had no idea how different this process could be. They didn't know that there was an Agile process with developer - customer interaction and iterations. They had no idea there was this test driven development thing. I don't know that they could fathom having a good experience working with programmers.
Micah had to break to them that they had been working with people doing it all wrong. He had to hit them with the bad news... but then he got to lift them with the news that a better way does exist, and that he knew what it was.
First, he went over, and it took quite a bit of time..., the list of things that were wrong with their current website's code. It was possible that the clients still thought it would be possible for 8thLight to work on the existing code base. Micah made it exceedingly clear that this was simply not reasonable.
Hahaha, there was one moment I recall when the clients proposed that 8th Light would take over the upkeep of the current website. Now, you must understand that the entire time Micah was presenting all the reasons why this code was awful, he had a rather stern and tensely maintained expression. It was pretty tough to tell what he was feeling, all you could see was that he was being honest. But when they proposed that he actually work on this code, it was as though all the muscles that had once been tense had suddenly gone limp and a wave of what can only be described as disgusted terror washed over him. They started laughing, and quickly after Micah regained his composure. The point was received.
After describing all the issues, and basically the reasons why it wouldn't be worth it for them to have anyone continue working on their existing code, Micah proceeded on to the good news.
This part was pretty amazing. Even though, and probably because, Micah remained just as calm and collected while presenting the good new, the clients were radiant with excitement. Here are these guys who have been waiting in Pergatory for a long time, and suddenly they see the light (the 8th Light ;-) ). And its not some preacher or enthusiastic peon overly excited to take them to a better place, but instead it is an experienced traveler who 'has been down both paths and will take them on the one less traveled by. And that made all the difference.'
The way I see it, these guys were in unfamiliar territory. They didn't know exactly what was right and what wasn't in this whole software world. They could only go on their instincts and their ability to decide who is worthy of their trust. And this is the point. It doesn't matter if you are selling the right product or the right idea. What matters is that your customer trusts you, and by that trust believes that YOU are right. The product could be flawless, but if you customer doesn't know much about it, and doesn't trust you, then the products perfection counts for nothing.
This is what Micah did. He wasn't trying to hype up 8th Light's methods and the Agile methodology. He wasn't trying to sound really excited about how excellent this manner of developing was. He was just being honest, straightforward, clear, and precise. He convinced them, through expertise and experience, that HE was right. It then follows that the ideas he was sharing with them were also right.
They, of course, got really excited. The clients began talking about how excellent it would feel to not have to wake up to bug reports. To be able to press one button, see a lot of green, and know that their program is working. Then there was the excitement for the future, and all the amazing things their website would do.
It seems that the key is to demonstrate an understanding, and to be clear and honest in order to build up a very important trust that is necessary for all business interactions.
They had a website with thousands of hits every week and nearly as many bug reports. The founder spoke about how he wished he could wake up in the morning and not dread all the emails he was going to have read about the many bugs people kept finding. They had gone through many developers and a few different companies, just trying to get any sort of website out there to start getting a fan base. After first having a group of developers fail, they found another group which got the website started, and then another group after that to try and maintain and slowly start getting rid of bugs.
These guys had no idea how different this process could be. They didn't know that there was an Agile process with developer - customer interaction and iterations. They had no idea there was this test driven development thing. I don't know that they could fathom having a good experience working with programmers.
Micah had to break to them that they had been working with people doing it all wrong. He had to hit them with the bad news... but then he got to lift them with the news that a better way does exist, and that he knew what it was.
First, he went over, and it took quite a bit of time..., the list of things that were wrong with their current website's code. It was possible that the clients still thought it would be possible for 8thLight to work on the existing code base. Micah made it exceedingly clear that this was simply not reasonable.
Hahaha, there was one moment I recall when the clients proposed that 8th Light would take over the upkeep of the current website. Now, you must understand that the entire time Micah was presenting all the reasons why this code was awful, he had a rather stern and tensely maintained expression. It was pretty tough to tell what he was feeling, all you could see was that he was being honest. But when they proposed that he actually work on this code, it was as though all the muscles that had once been tense had suddenly gone limp and a wave of what can only be described as disgusted terror washed over him. They started laughing, and quickly after Micah regained his composure. The point was received.
After describing all the issues, and basically the reasons why it wouldn't be worth it for them to have anyone continue working on their existing code, Micah proceeded on to the good news.
This part was pretty amazing. Even though, and probably because, Micah remained just as calm and collected while presenting the good new, the clients were radiant with excitement. Here are these guys who have been waiting in Pergatory for a long time, and suddenly they see the light (the 8th Light ;-) ). And its not some preacher or enthusiastic peon overly excited to take them to a better place, but instead it is an experienced traveler who 'has been down both paths and will take them on the one less traveled by. And that made all the difference.'
The way I see it, these guys were in unfamiliar territory. They didn't know exactly what was right and what wasn't in this whole software world. They could only go on their instincts and their ability to decide who is worthy of their trust. And this is the point. It doesn't matter if you are selling the right product or the right idea. What matters is that your customer trusts you, and by that trust believes that YOU are right. The product could be flawless, but if you customer doesn't know much about it, and doesn't trust you, then the products perfection counts for nothing.
This is what Micah did. He wasn't trying to hype up 8th Light's methods and the Agile methodology. He wasn't trying to sound really excited about how excellent this manner of developing was. He was just being honest, straightforward, clear, and precise. He convinced them, through expertise and experience, that HE was right. It then follows that the ideas he was sharing with them were also right.
They, of course, got really excited. The clients began talking about how excellent it would feel to not have to wake up to bug reports. To be able to press one button, see a lot of green, and know that their program is working. Then there was the excitement for the future, and all the amazing things their website would do.
It seems that the key is to demonstrate an understanding, and to be clear and honest in order to build up a very important trust that is necessary for all business interactions.
Subscribe to:
Posts (Atom)
