Girish Juneja interviewed at Strata Santa Clara 2013

Thank you for inviting me.

You know, Hadoop is an interesting framework. It involves a combination of storage, network, and compute. As we started working on Hadoop about two and a half, three years ago, we found there were so many different ways we could optimize the overall infrastructure to perform better in a particular cluster by optimizing at those layers. So to give you a few examples, Intel provides SSD solid state drives. Now, to use SSD effectively in a Hadoop environment, you need to do some software-level caching. So we have that unique knowledge to bring the caching element with SSD to optimize the platform. Another example that you may have heard: we talked about our Project Rhino. A small part of that is using AES-NI instruction set that are there in every Xeon processor to be able to encrypt the HDFS and edge-based data. And we have a very clear view on how that optimization can be delivered. So it was far easier for us to deliver that optimization into the open source than it would be for somebody else. So, if you look down the road and we see hardware roadmaps a couple of years down the road, there's a fundamental shift in data center coming where you stop thinking of the data center unit as a server and you start to think of it as a rack or as even a larger environment. And in that, how do you change the Hadoop infrastructure or evolve it so it uses that framework more effectively becomes something that Intel does very well, and that's why our role in providing a distribution and then supporting it and putting the improvements into the open source to drive that forward.

That's, you know, part of our reason to become a participant in this ecosystem as opposed to a pure technology enabler, is that the framework itself needs to expand to address different kinds of needs being asked from it. Right, so it started off with MapReduce paradigm. You're hearing a lot about SQL, we hear a lot about text search. You also hear the layer above it which is how to pull all these different kinds of data science, OLTP, the transaction querying, and analytic coding together in a way that different audiences can use it. And so our view is this framework needs to evolve so Hadoop becomes the common substrate on which data scientists, the online transaction coding, different kinds of processing can happen. And not all of it will happen within Hadoop; some of it will happen outside. And you know we talked about our collaboration with SAP at the launch event. That's an example of how the real-time querying happens outside of the framework but it integrates nicely. So if you want to do an OLAP query that uses a lot of data that is not something that will fit in memory, you could use Hadoop and Hana and SAP Hana together to respond to those kind of queries.

Yeah, it's pretty fascinating. You know, we work at Intel, we work very closely with many of the ecosystem partners that are providing capability of generating new data -- data that didn't exist just a year ago or even six months ago. To give you an example, there is a pilot that's running in Texas where they're collecting smart meter information from houses and doing analysis on that data so that they could point exactly to what equipment in a house might be sucking too much power and maybe ready for replacement because there's a more efficient way to do it. So that's an example of new data. Right, automobiles are generating – they have multiple processing cores now and they are throwing tremendous amount of data as they move around in traffic. And how do you analyze it to deliver better data to the automobile manufacturer so they can design better machines down the road? So those are examples of new data. I mean, dark data is data that actually existed but because it was either too large or stored in places it couldn't be accessed, it was not being used. So an example of that I would just point to is all of the construction data of all the buildings in New York, for example. The customer pilot we work with has data on every knob, every window that is in Manhattan buildings. So now, with using Hadoop framework, we, working with them, have designed an application where they could go to an architect who has the job of setting up a new building in Manhattan and say, 'Well, your building is facing southeast, the wind shear is so much, this is the kind of sunlight you want to have inside, here are the specifications of different pea spots you can put inside the building.' So that's using dark data more effectively to solve tomorrow's problems.

So, you know, again, a lot of Hadoop ecosystem kind of evolved from environments where speed and performance was important, but also being able to find meaningful data out of these users' information -- Facebook, Yahoo web environments -- was important. But as Hadoop moves from that framework into getting into places like healthcare, or government, or financial services, the security elements become quite critical, compliance becomes important. So, you know, who has access, who accessed this particular data last time, if you look at HIPAA compliance or PCI DSS compliance, you have to be able to predictably decipher and tell who accessed certain amount of data. You have to be able to encrypt individual piece parts of data. You have to be able to provide access control so you can control in a shared environment like a cloud service provider that if I have two competing vendors whose data I'm co-hosting in my framework, they're not able to look at each other's data. So that's an example of security need coming up as we move into more enterprise-ready environments and out of pure web environments.

Thank you. Thanks for having me.