Yeah, which means it isn't used by default.
So the broadcast is kind of a slightly more advanced thing. It's useful when you have a job where you need, say, a large parameter vector you're learning and you need to send it to all the nodes. Turns out just copying it from the master to each node is very slow. And so we have a bitTorrent-based broadcast and other things like that. Yeah, I wasn't going to talk about that, but that's actually something we might talk about at another meeting. That's one of the things we did to make this fast for machine learning.
Yeah, cool. Okay, so let's just do a more interesting file also. So I have here a file in HDFS on this cluster. This is a Wikipedia text file. And I think some people may have seen this kind of demo before, but this is the text file we'll be using in this job. So this is a file sitting in HDFS and it's about 50 GB large. So how do you work with a bigger file? You can do even if the file is big there are still some things you can do in Spark to explore it interactively. And that's one of the things that makes it cool for actually working with this data.
So, for example, I'm going to call first here. First returns the first element of a data set. And you saw that was really fast. So, first in this case just opens the file and reads one line. It doesn't try to do anything fancier than that. So this file here is a tab-separated values file and you can see this is the line that we read. And you can see it has sort of five fields. It has an article ID. It has the title of the article. This is maybe the alphabetically first thing. It has the date when it was modified. It has the XML and you don't see the last field but there's also a plain text field. Actually, if you want, we can do that.
Yeah, let's split it and let's look at the last piece. So this is a funk pack. Okay. So this is how you can use this to sort of play with data interactively. And of course, we can also play with the whole data set by doing stuff on it in parallel. So let's just show some of the stuff we can do with this. So let's do the first thing, let's just count how many lines there are or how many articles.
Oops. Yeah, it's not called file. It's called wikiFile. Okay. So this is now running on the cluster. And this is not using the in-memory data. It's just this on-disk data.
Saying managed library, managed directory.
Oh, oh, interesting. I think I've seen sometimes that problem. If you have, it helps to delete your .m2 and .ivy2, which I sometimes, I think there's one library we depend on that the old version in Maven was wrong and it confuses things, but I don't know, I haven't, maybe it's not the same thing.
Yeah. Yeah. Okay. Yeah. So this was working with the on-disk data and we did a count and we see there were this many articles in this dump of Wikipedia.
Yeah, so this is how you can work with data this way. Working with data when it's on disk is slow though. So, one of the cool things you can do in Spark is of course load it into memory. And I'll just do that before going back to show you some of the other things.
So let's actually do, as you saw looking at this thing, it's not really easy to work with a string like this. So we're actually also going to define a class to represent these articles and we'll have the title and text of the article only. And let's do a map function to pull those guys out of the file. So this is actually going to be a couple of functions here.
So, first we're going to take each line of the file and split it by tabs. This is the syntax you can do for that in Scala. So this is just shorthand for the function like x => x.split. You can also write it that way. But if you want to write it a bit more concisely, you can use underscore instead of x. And what we're going to do here, some of the articles don't actually have the plain text field. So we're going to filter to the ones that have exactly five elements. And now we have these arrays of five elements and we're going to do a new article and we'll take field one and field four.
So this now we have a data set of articles. The other thing I'm going to do here is tell the system to cache these ones in memory. So the first time it computes them it's going to take a while but after that they'll be in memory. And now we can do stuff like say articles.count. This is going to be the ones that didn't have, that had exactly five fields. So it'll be a little different. So let's just do that. And so you can see now as it's running, it's adding these cache entries.
I should also show you what this cluster looks like in the meantime. So this is kind of... oh, that's taking... yeah. So this is the Mesos cluster we're on. And you can see I'm running the Spark shell and it has this much memory. It's using this many CPUs and I have a bunch of nodes.
