I'm broken (node.js)

[tourmaline]

I don't know what is wrong with me, but I cannot grasp async programing. I think I'm about ready to completely throw in the towel. I've been studying and reading and trying and studying, and I can't get it. I've been on some IRC channels getting help, and each time I go to perform a simple async action, it doesn't work because my head is not async.

I went over to nodeschool.io, and I'm stuck on exercise 4. I cannot make it happen. I've been programing for 15 or more years, but I cannot do a simple async exercise.

My main contention is how I have my game plan laid out for my game, which was done way before any code. My first move with whatever language I will use, will be:

3 functions: 1,2,3. 1 calls 2, 2 calls and sends result to 3. 3 returns result to 1, where 1 will now do something else (porbably interface with WEBGL). 1 will not always call 2, maybe 1 will cal 3, or we start at 2 and go to 3 based on what happened the last iteration, but you see where I am going.

Everyone says NO. But I can't help it. It's the way I code, for years. I'm getting frustrated, because I want to write some code. I think my brain is too old to change my way of thinking.

[hallsofvallhalla]

You are over thinking it. I make it real simple in my videos.

https://www.youtube.com/watch?v=mSE6xHkcX0w
https://www.youtube.com/watch?v=IbvfZ8MuO-0
https://www.youtube.com/watch?v=EQ_kenymItI
https://www.youtube.com/watch?v=hOEyVhesl4k
https://www.youtube.com/watch?v=MUsp0lriV8k
https://www.youtube.com/watch?v=7Kqmd0Je9XM

[Jackolantern]

That is what I am thinking, too. If you have seen code where function 1 calls back 2, and then 2 sometimes calls back 3, and on and on, either that person doesn't know node and was screwing about, or they are highly advanced. Kind of odd that it breaks down that way, but that is the way it is. 98% of node code out there doesn't work that way. You just write most of your code as normal, and when you need to do any kind of I/O, you call a special async method and any subsequent code that relies on results from that I/O needs to go into the callback method that is traditionally passed in as the final parameter of the async method call. That is really all you need to know (that and after the async method call is made, V8 will continue on down your code and will pop back up to the callback when the I/O call returns).

There are only a couple of real tricks you need to learn to deal with the vast majority of situations. First, if you have a major branch in your code with one or both sides of the branch relying on callbacks (meaning they will finish in an unknown amount of time in the future), it may seem like you need to write the exact same code twice to finish up your procedure. But of course the solution here is going way back to your procedural days: just write the end of the procedure in a method and call it where it is needed.

Another common pattern you will see outside of this is when you make a regular method call, and inside of that method body you have to make an async call. You see, this would cause a problem, because in the middle of the function body you called, V8 would suddenly jump back to your method, which likely wouldn't have what was needed to continue. You can solve that this way:

Code: Select all

function someFunc(obj, cb) {
     //do some stuff, and need to call an async method and send in obj variable
     asyncCall(obj, function(returnVal) {    <--this would be a true async I/O call
          //now you have the value you want to return, but you can't return in
          //because at this point, the original method call (someFunc) has actually 
          //already returned O.o  
          //instead, you will call that callback method you sent in here (cb)
          cb(returnVal);
     });
});

//your main code
var str = "Hello!";
//now call your method, which because it has an async call in it and takes a callback as an argument, is basically itself async 
//(even if that isn't quite right; you can't write true async methods yourself in node, because the actual async I/O stuff is happening
//   ...in the C bindings underneath V8)
someFunct(str, function(valueReturned) { 
     console.log(valueReturned);
});

I have heard some people call this "volleying". It boils down to realizing your function call body will contain an async call, so you can't expect a normal return value. Instead, you have to pass in a callback function and handle it the "node way".

Those two things took me a while to figure out, but aside from that, it is all pretty straight forward once you wrap your mind around callbacks. And simply keep this in mind: "Node will never wait for you". Always keep in mind it is always racing ahead, and anything that will take time, node will already be a mile or two down the road before your callbacks even return with the I/O results. You have to code with that in mind.

I hope this may help just a bit

[tourmaline]

Thanks for the replies!

The videos are cool Halls; thanks for linking them.

Any async operations are part of node then. Just passing a function as an argument does not make a method async, yes/no?

And if that is true, what about something simple like fs.readFile and fs.readFileSync. What would be the detriment to using the sync version?

I don't see where the process running ahead, and then trying to give a result back to the process at a later point makes any sense. I'm apparently not seeing all the benefits to asynchronous programing. Ok, updating character positions in a game, players in game, environment changes in a game; asynchronously, that makes sense. Broadcasting to 100 clients, yes, asynchronous. But I don't see any benefit to reading a file. I don't see a benefit to making a DB query asynchronously. I always want to use the data at the very moment I am querying it; not down the road sometime. If I insert data, I don't want to let the program churn on before I've verified the insert.

As I read, I'm finding hints about how node might work: single pipe that all clients connect to and just beats on. Ok, so having node at the end of the pipe chucking requests to where they go, all at the same time, sure. But in the end, how is that better than a separate pipe for each client? It seems like a problem for the sake of a solution.

I've been through the manual quite a bit, and I'm finding some pretty neat things. But in practice, I'm stuck on fs.readFIle, a callback, and how that is practical to me (or anyone for that matter). I keep trying to take that example, and expand on it to how I would use it, and it just fails.

EXAMPLE: Instead of one file to read, read 3. give each file result back to it's calling function where it appends them to a local variable. Now give that result back to an initial function that initially decided to grab those files or not (among other functionality), based on the presence of that readfile string being present, or a set bool. It doesn't need to be 3 functions; but if I want to separate string cleaning and concat away from the logic of deciding to read the file, and a 3rd function that just performs the file read. 3 functions to get there is not completely far fetched. It's far better than a 3 level callback nest, if I could even think through such a thing.

I suppose I should look at something else for a minute, maybe start looking at WEBGL so I have an interface to apply things to. Maybe build a log file scheme with that. Maybe some of this node beginner stuff without input is a drag (thanks Halls for helping me notice a little bit outside of where I'm stuck).

[Jackolantern]

Any async operations are part of node then. Just passing a function as an argument does not make a method async, yes/no?

No, just passing a function as an argument does not make it async. If you did this:

Code: Select all

function notAsync(a, b, callback) {
     callback(a + b);
}

notAsync(1, 2, function(result) {
     console.log(result);
});

There is nothing async about this. What I mean by "async" to be specific is when execution will go ahead and do something else while it is waiting for the true async function call to return. But that doesn't happen here. This code would executed procedurally, and nothing else would be done from the time it starts to the time it ends.

And if that is true, what about something simple like fs.readFile and fs.readFileSync. What would be the detriment to using the sync version?

Sync (or "blocking") versions of I/O operations being included in node is controversial, because you should never call them in any kind of server that is handling concurrent users. Doing so would kill the performance of your server. That is because the sync versions of the I/O functions halt the program until the result is returned, just like what happens in Java, C++, and most other languages when you make normal, blocking I/O calls. That is why those traditional languages resort to multithreading for concurrency. Multithreading is far more complex and requires much more planning than asynchronous programming.

The sync versions are mostly included for people to create small, single-purpose tools and utility scripts in node. But what is controversial is that in no case are they ever required. Those small utility scripts can still use the async versions, but I guess when you don't need concurrency, blocking I/O is more simple.

I don't see where the process running ahead, and then trying to give a result back to the process at a later point makes any sense. I'm apparently not seeing all the benefits to asynchronous programing.

The benefits of asynchronous programming become much more apparent when they are matched with an "event loop", which node is. For example, in a real-time website application, you have events set up to serve various webpages, and you may also have Socket.io events set up to handle socket connections, socket messages, etc. When a request for a webpage comes in, the code inside the event will make an async call to the file system to retrieve the webpage code. Now since it is async, it doesn't wait for the slow hard drive to retrieve the webpage. It is free to go and begin servicing other website requests, Socket.io events, etc. And when the system is notified that the website code has arrived from the hard drive, possibly 3 or 4 events were processed in the meantime, but node jumps back to send the webpage to that user.

A good metaphor I read somewhere is like if the server was a waitress and the requests it is receiving are like restaurant customers. How many customers could a restaurant serve if the single waitress could only wait on one table at a time? Almost the whole time would be wasted while the waitress just sits and smokes a cigarette and watches her one table eat, all the while other customers are staring at the walls waiting for the waitress. The only way to fix that would be to hire more waitresses, who will all also sit and smoke cigarettes while their one table chooses what they want to order, eats, etc. That is the same solution as multithreading. Most of the time each thread uses is wasted waiting for I/O to be performed (some estimates put it around 98 - 99% of its time). However, the async solution would be more like restaurants actually work: Since each customer requires such a small amount of actual time interacting with the waitress, she handles many tables at a time by helping the other tables while the customers choose what to order, eat, etc.

The actual figures are staggering as to what takes what amount of time in a modern computer. Here are some figures:

CPU executes a typical operation (ADD, SUB, MOV, etc.): 1 nanosecond
CPU executes more complex operation (MUL, DIV): 2 - 3 nanoseconds
Fetch from L1 cache: 0.5 nanosecond
Fetch from L2 cache: 7 nanoseconds
Fetch from main memory: 100 nanoseconds
Send 2K bytes over 1Gbps network: 20,000 nanoseconds
Read 1MB sequentially from memory: 250,000 nanoseconds
Read 1MB sequentially from disk: 20,000,000 nanoseconds
Send packet from US to Europe and back: 150,000,000 nanoseconds

As you can see, the CPU, cache and even the main memory are orders of magnitude faster than the disk and network. So you can see that I/O is what is holding back any computer from the true speeds its processor can attain. So when you make those disk and network operations non-blocking, you are casting off that bottleneck. That is why a single-threaded node server can keep up quite well with an 8-core server running with multithreading. Because that single-thread is only dealing with the super-fast CPU, cache and main memory operations. Disk and network access and sequential memory access are all handled asynchronously since they are very slow.

Ok, updating character positions in a game, players in game, environment changes in a game; asynchronously, that makes sense. Broadcasting to 100 clients, yes, asynchronous. But I don't see any benefit to reading a file. I don't see a benefit to making a DB query asynchronously. I always want to use the data at the very moment I am querying it; not down the road sometime. If I insert data, I don't want to let the program churn on before I've verified the insert.

To the computer the result comes back later, but it still happens so fast a human can't comprehend any pause. Remember, we are talking about nanoseconds here. And whether or not your call is async or sync, it still takes about the same amount of time to retrieve the data. The wait isn't in node doing something else. It is in the much longer wait time on I/O versus what the CPU can perform. There can be some unpredictability to what order events can occur, but that is the way event-based programming always happens. You can't tell your users what order they need to press buttons in, or who needs to request a page first, so it doesn't really matter to event-based applications. And that tends to be how networked servers operate anyway, async or multithreaded.

As I read, I'm finding hints about how node might work: single pipe that all clients connect to and just beats on. Ok, so having node at the end of the pipe chucking requests to where they go, all at the same time, sure. But in the end, how is that better than a separate pipe for each client? It seems like a problem for the sake of a solution.

Because having a separate pipe for each client requires multithreading, and everyone pretty much agrees multithreading sucks. That is because to multithread, you are bowing down to the almighty power of our lord "The Thread Scheduler". The thread scheduler is completely controlled by the OS. It decides when to switch threads, which one gets executed when, etc. And because of that, debugging multithreaded servers is an incredible pain. An error may only occur 1 every 100 times, or only when a completely unrelated program is running in the background at the same time, or only when a certain number of threads are executing. The thread scheduler is an invisible and untouchable, yet highly important part of your application. Once you have dozens of threads interacting, when something breaks, put on some coffee and prepare to pull your hair out.

I've been through the manual quite a bit, and I'm finding some pretty neat things. But in practice, I'm stuck on fs.readFIle, a callback, and how that is practical to me (or anyone for that matter). I keep trying to take that example, and expand on it to how I would use it, and it just fails.

Honestly, I have only had to read a file once or twice. I would suggest to get ExpressJS from npm, set up a project and do some tutorials in that. Everything would likely make a lot more sense in that environment. That is how I got the hang of node. Then add Socket.io, and try making a simple web-based chat room. That is exactly the order I went in.

But as far as what you mentioned, fs.readFile is an async method to read an entire file. The callback function you define for the async call will receive the contents of the file as one of its parameters. I have only seen it used for writing web servers to get HTML files, which you likely won't be doing much (most good web frameworks use Connect as web server middleware).

EXAMPLE: Instead of one file to read, read 3. give each file result back to it's calling function where it appends them to a local variable. Now give that result back to an initial function that initially decided to grab those files or not (among other functionality), based on the presence of that readfile string being present, or a set bool. It doesn't need to be 3 functions; but if I want to separate string cleaning and concat away from the logic of deciding to read the file, and a 3rd function that just performs the file read. 3 functions to get there is not completely far fetched. It's far better than a 3 level callback nest, if I could even think through such a thing.

Nested callbacks are entirely common and accepted in node. Trying to resist them leads you to some weird patterns, or, if you really don't like nested callbacks, to the "promises" pattern/libraries. The latter is how C# handles asynchronous programming.

I suppose I should look at something else for a minute, maybe start looking at WEBGL so I have an interface to apply things to. Maybe build a log file scheme with that. Maybe some of this node beginner stuff without input is a drag (thanks Halls for helping me notice a little bit outside of where I'm stuck).

I highly suggest to start off with Express and Socket.io. That will likely be the skeleton of your server when you move on to working on a WebGL online game anyway. Things are much clearer when working with the web to start off with. Perhaps just trying to do it with no output is what is really causing you problems, kind of like trying to learn practical PHP from only the command line tool.

Hope this helps!

[tourmaline]

Thank you for such a concise response! I haven't dealt with multi-threading issues, nor have I ever worked with an event loop.

One thing I am taking away from this right now, is that async is not about one person or one request, but how the application operates as a whole. I've never started a project thinking how many requests are going to stack up and be processed, but rather how one user is going to move through the process I am designing. There is a little practical concern for async at that stage. But I now see where I am broken.

As much as I am against it, maybe I'll go back and try express again. I had read that express 4x was changing some fundamentals, and I had difficulty installing the 4x rc, but now I have it installed. I'm against layering helpers upon helpers, without having a thorough mastery of what the helpers are saving me from doing myself. So I almost save time by not using such things, but I should just go with the flow or I'll never get started.

I must be getting old; I'm starting to sound like a person I may have laughed at years ago.

Thank you again!!!

[Jackolantern]

Node is kind of odd in how it can sometimes be easier to learn it by going to a higher level with a framework first, let asynchronous programming sink in, and then if you need to you can go back lower again. For example, if you wanted to create your own framework.

As far as programming thinking about only one user, that is definitely not what node was designed for. It was specifically designed to ease the development of networking applications, in particular servers

[tourmaline]

As far as programming thinking about only one user, that is definitely not what node was designed for. It was specifically designed to ease the development of networking applications, in particular servers

You know what I mean though. As far as real world solutions, who thinks of stacks of user requests? I think of what the user wants and how to get the user to what he/she wants, one user at a time.

I hope I can get this soon. And thanks again!

[Jackolantern]

Your most welcome