I have a new blog at adamhurwitz.net on Security, BigData, and the Cloud.
You can also follow me on Twitter @adam_hurwitz
Friday, May 20, 2011
Wednesday, March 9, 2011
The Armory Show NYC
I visited The Armory Show last week-end, a very popular art fair (now too popular for some, of course) that draws galleries from all over the world to show off their contemporary works. They've added some modern work, but the contemporary is still the focus.
And as I walked along I couldn't help but think that there are artists ("artists") basically trying every possible idea of what could be considered art. It brought to mind a depth-first search algorithm, as if some huge computer cluster were exploring every idea and then every variation of every idea relentlessly looking for art, or at least something that people will buy.
And as I walked along I couldn't help but think that there are artists ("artists") basically trying every possible idea of what could be considered art. It brought to mind a depth-first search algorithm, as if some huge computer cluster were exploring every idea and then every variation of every idea relentlessly looking for art, or at least something that people will buy.
Sunday, January 30, 2011
What is BigData?
At some point quantity becomes quality. You can add a few more of something and suddenly it can become something different. For data we have crossed that threshold again over the last decade. Now whether that point is defined as terabytes or petabytes is in-itself relatively unimportant. What is important is that people have had to manage this qualitatively different amount of data, and in order to do so they have created new technologies, new techniques, and new ways of thinking about and analyzing the data. This in turn, of course, has created new opportunities. This new paradigm is known as BigData.
Practically when people speak about it, I believe they are referring to two specific things: 1. the new set of data management and processing technologies that involve either distributed or parallel computing; 2. an exploratory Business Intelligence activity that utilizes machine learning and statistics to extract knowledge from the data.
Practically when people speak about it, I believe they are referring to two specific things: 1. the new set of data management and processing technologies that involve either distributed or parallel computing; 2. an exploratory Business Intelligence activity that utilizes machine learning and statistics to extract knowledge from the data.
Monday, December 13, 2010
New Date Announced for NY Tech Mixer Meetup
If you're interested in the NYC start-up scene - and these days who isn't? - then you have to come to the next Demos and Drinks on January 18. At Demos and Drinks a handful of start-ups get to demo and everyone drinks and socializes. The crowd is a mix of techies, designers, biz folks, and investors. It's organized by the NY Tech Mixer Meetup which has over 1500 members and regularly attracts 300+ people per event. Check the event page for more details. Sign-up now before it sells out. It's going to be a great event and it's going to be monthly from now on.
Tuesday, November 23, 2010
O'Reilly Strata Conference on Big Data
I'm really honored to have been chosen to be part of the O'Reilly Strata Conference, coming up in February in Santa Clara. It is a new conference that focuses on the business and practice of data.
All industries are experiencing an explosion of data volumes. But most companies are finding it hard to keep up and make use of all the data that they generate or have access to. This is creating great opportunities for those who understand how to effectively collect and use large quantities of data.
I will be part of the Real World Applications Panel: Enterprise and Industry. My focus will be the legal industry, where I work now, and there will be two other panelists from the music and meteorological industries.
It is going to be a great event. Drop me a line if you are going to be there.
All industries are experiencing an explosion of data volumes. But most companies are finding it hard to keep up and make use of all the data that they generate or have access to. This is creating great opportunities for those who understand how to effectively collect and use large quantities of data.
I will be part of the Real World Applications Panel: Enterprise and Industry. My focus will be the legal industry, where I work now, and there will be two other panelists from the music and meteorological industries.
It is going to be a great event. Drop me a line if you are going to be there.
Friday, October 15, 2010
About Us is the most important
I was looking at the web stats of a company website the other day. The company is relatively small and provides services to businesses. And the thing that struck me right away is that the most popular pages were the About Us pages.
This made me realize that when I look at other company websites, I instinctively go straight to the About Us pages instead of the products and services pages. If I'm on their website, it's more than likely that I already know what they do. And if I don't, then all I need is a quick scan of the homepage. I'm really much more interested in seeing the management team, the board of directors, the investors, and maybe some history in order to understand the company. Only after I get a sense of who's involved do I consider actually reading about their services or products. I have a feeling that this is becoming the norm.
This made me realize that when I look at other company websites, I instinctively go straight to the About Us pages instead of the products and services pages. If I'm on their website, it's more than likely that I already know what they do. And if I don't, then all I need is a quick scan of the homepage. I'm really much more interested in seeing the management team, the board of directors, the investors, and maybe some history in order to understand the company. Only after I get a sense of who's involved do I consider actually reading about their services or products. I have a feeling that this is becoming the norm.
Sunday, August 29, 2010
HCIR 2010
I attended the HCIR 2010 workshop last Sunday on “Bridging Human-Computer Interaction and Information Retrieval”. Dan Russell from Google gave the keynote. This is a guy who is passionate about what he does, and it seems that his job is to study how people search, including how they think about searching, how they conduct searches, how they use search interfaces, etc. and to figure out how they could do it better.
One of the biggest problems that exists, he believes, is that there is no training or education for search. The overwhelming majority of people just do not know how to search well. For instance, he found that most people don't know that you can use control-F on almost every program in order to find a word or phrase. This is something that most techie people take for granted when they are looking for something. For certain searches, you look at the results in Google, you pop open one of the results that looks promising, and then you control-F to find your keyword on the page and see if it is what you're looking for. The step of using control-F obviously speeds things up greatly. According to Dan this is something that most people just don't know about. I was skeptical and so I mentioned it to someone I know who is a professional (college educated, works in an office, etc.) and he knew that you could do that in Excel but didn't realize that it worked in all programs. I was sort of astounded by this, which was something that Dan had mentioned happens to him all the time. So for search instead of making the results somehow better, there may be a lot more value in training the user and providing tips or heuristics to help the user while they are searching.
One of the biggest problems that exists, he believes, is that there is no training or education for search. The overwhelming majority of people just do not know how to search well. For instance, he found that most people don't know that you can use control-F on almost every program in order to find a word or phrase. This is something that most techie people take for granted when they are looking for something. For certain searches, you look at the results in Google, you pop open one of the results that looks promising, and then you control-F to find your keyword on the page and see if it is what you're looking for. The step of using control-F obviously speeds things up greatly. According to Dan this is something that most people just don't know about. I was skeptical and so I mentioned it to someone I know who is a professional (college educated, works in an office, etc.) and he knew that you could do that in Excel but didn't realize that it worked in all programs. I was sort of astounded by this, which was something that Dan had mentioned happens to him all the time. So for search instead of making the results somehow better, there may be a lot more value in training the user and providing tips or heuristics to help the user while they are searching.
Sunday, July 11, 2010
Hiring - finding out what you're really looking for
I've spoken to a few people recently about hiring. They were in an initial stage where they've said to me that they needed to find someone for a tech position. And in each case when I've asked for detail about the profile needed, it turned out that the person was not that clear on what they exactly needed. Everyone who has done hiring has been in this situation. You know you need someone and think you know what you need, but as soon as you start looking, you start to realize that you're really not sure.
Obviously it would be best to know before you start talking to candidates. For many hiring managers, though, it seems that they start by browsing ads posted by other companies to make a profile. And then the process of interviewing people is the way that they figure out what they are actually looking for.
One easy and obvious way to get a handle on this is to do the job yourself for a little while. Of course this depends on the job and company, but if you can, you'll clearly get the best sense of what's needed. This of course happens naturally in small, growing companies, but isn't always an option.
Another thing to do is focus on creating a good practical test for the job. For technical positions this is a necessity. And I believe that the best practical test simulates real work as opposed to theoretical problems or logic puzzles. So when you put together the test for what the candidate needs to know, you should be reviewing real work problems and tasks.
These are two things that have worked well for me. I also try to keep in mind that when you don't really know what you're looking for, not only can you make negative impressions on candidates, but - worst of all - you may not realize it when you find the right person.
Obviously it would be best to know before you start talking to candidates. For many hiring managers, though, it seems that they start by browsing ads posted by other companies to make a profile. And then the process of interviewing people is the way that they figure out what they are actually looking for.
One easy and obvious way to get a handle on this is to do the job yourself for a little while. Of course this depends on the job and company, but if you can, you'll clearly get the best sense of what's needed. This of course happens naturally in small, growing companies, but isn't always an option.
Another thing to do is focus on creating a good practical test for the job. For technical positions this is a necessity. And I believe that the best practical test simulates real work as opposed to theoretical problems or logic puzzles. So when you put together the test for what the candidate needs to know, you should be reviewing real work problems and tasks.
These are two things that have worked well for me. I also try to keep in mind that when you don't really know what you're looking for, not only can you make negative impressions on candidates, but - worst of all - you may not realize it when you find the right person.
Wednesday, June 23, 2010
Least amount of process principle
The least amount of process needed to do something is the right amount of process. For me this is a management principle that is akin to Occam's razor in science, where the simplest explanation is the correct one. As obvious or simple as this principle may be, it is violated all the time.
My main concern is with software development, but this principle is more universal for management, especially of creative work. The reason why this principle works is that all processes are carried out by a unique set of people who are all unique in their skills, personalities, and characters. As such, too much process will stifle the group and not allow the individuals to emerge and achieve.
This also means that you will encounter real difficulties when you take a defined process and try to implement it, because a defined process relies on roles that people have to play. You may get a group to enact a defined process, everyone playing their roles, but with time the process and roles will evolve based on the individuals. Of course this customization of the process is a good thing if you understand what's happening, and don't try to force things back to the defined process.
My main concern is with software development, but this principle is more universal for management, especially of creative work. The reason why this principle works is that all processes are carried out by a unique set of people who are all unique in their skills, personalities, and characters. As such, too much process will stifle the group and not allow the individuals to emerge and achieve.
This also means that you will encounter real difficulties when you take a defined process and try to implement it, because a defined process relies on roles that people have to play. You may get a group to enact a defined process, everyone playing their roles, but with time the process and roles will evolve based on the individuals. Of course this customization of the process is a good thing if you understand what's happening, and don't try to force things back to the defined process.
Wednesday, May 19, 2010
Kanban vs Scrum
I recently came across a great free book written by Henrik Kniberg and Mattias Skarin called, "Kanban and Scrum - making the most of both". It includes the most concise description of each development process that I've seen.
For Scrum, your overall concern is with timeboxing everything, making sure that there are short cycles and short meetings that are strictly adhered to. With Kanban, your main concern is with limiting the amount of work in progress. By limiting how many items are in each phase of development, your aim is to establish an on-going flow. It seems to me that Scrum is better geared for new product development and Kanban is better for existing products that need maintenance and new features. There is obviously much more to each and I recommend you read their book.
For Scrum, your overall concern is with timeboxing everything, making sure that there are short cycles and short meetings that are strictly adhered to. With Kanban, your main concern is with limiting the amount of work in progress. By limiting how many items are in each phase of development, your aim is to establish an on-going flow. It seems to me that Scrum is better geared for new product development and Kanban is better for existing products that need maintenance and new features. There is obviously much more to each and I recommend you read their book.
Monday, February 15, 2010
Business Value is Always the Goal
When you are creating software for a business, the creation of business value is always the goal. In general there are only two ways that business value is created. Either you are saving money because you make a process more efficient or you create new revenue with a new process, product, or feature.
As the goal of software development, it means that we are not even considering the qualities of systems that software developers are generally concerned with. So it makes absolutely no sense to create a system that is really easy to maintain if that increases the cost of the system so that it doesn't provide business value. These qualities have to be understood as constraints and they are understood in terms of trade-offs. If you are absolute about them, then you will have a hard time creating a successful system. This seems completely reasonable and obvious, but it happens all the time. Always remember: the cost of the system has to be less than the business value created.
As the goal of software development, it means that we are not even considering the qualities of systems that software developers are generally concerned with. So it makes absolutely no sense to create a system that is really easy to maintain if that increases the cost of the system so that it doesn't provide business value. These qualities have to be understood as constraints and they are understood in terms of trade-offs. If you are absolute about them, then you will have a hard time creating a successful system. This seems completely reasonable and obvious, but it happens all the time. Always remember: the cost of the system has to be less than the business value created.
Thursday, December 17, 2009
The Daily Scrum or Morning Meeting
A scrum or morning meeting is the best way to get control of a development process. Having it in the morning seems like the most natural part of the day for everyone to synchronize, obviously, before everyone has started working. Not first thing in the morning because everyone needs time to have some breakfast, review new email, read the news, and get settled in. About a half-hour should do it. The other good thing about having it in the morning is to ensure that everyone gets to work on time, which gets to the essence of how a scrum works - peer pressure. When someone is late to the meeting, it is not just clear to their manager, but to the whole team. So they are not letting their manager down, they are letting the team down.
During the meeting, each member of the team goes over what they have accomplished since the day before and what they are working on. According to strict principles, the meeting is not supposed to become a status meeting run by the project manager (technically the role is supposed to be called a scrum master, but I don't like that name). Personally I like having the project manager run the meeting. I prefer the direction that it gives. And you still end up with the peer pressure because everyone is listening.
During the meeting, each member of the team goes over what they have accomplished since the day before and what they are working on. According to strict principles, the meeting is not supposed to become a status meeting run by the project manager (technically the role is supposed to be called a scrum master, but I don't like that name). Personally I like having the project manager run the meeting. I prefer the direction that it gives. And you still end up with the peer pressure because everyone is listening.
Sunday, December 13, 2009
Using Humans in Your System Design
A mistake that a lot of people make when creating software is to try to automate everything. When you are making software for a business, you are generally automating a process and there is often a desire or drive that people have to make software that automates the whole process. They get into an all-or-nothing frame of mind that, unfortunately, usually ends up with nothing getting deployed, or, even worse, software that gets deployed which tries to do it all and fails, disrupting the business and wasting a lot more money.
There are many different strategies and methodologies to deal with this situation. Agile is probably the most popular and successful. One of the main tenets of Agile is to deliver business value as soon as possible. And in most situations in order to deliver that value quickly, you obviously can't program everything. And if you can't program everything, you are going to have to learn how to design humans into your system. This is something that a lot of software people seem uncomfortable with because they view the system they are building purely as a piece of software and think that automation is the ideal. But full automation is not necessarily the goal, creating business value is. (I'll have more to say on this in a later post.)
Saturday, October 24, 2009
Computer Forensics Paper
Forensic Focus, the leading computer forensics community site, has posted one of my papers, titled, "Simple Steganography on NTFS when using the NSRL." It's a relatively simple idea, but important for computer forensics investigators who use the NSRL (National Software Reference Library from NIST). For those of you who aren't familiar with the NSRL, it contains the hashes of millions of files from operating systems and applications. This information is then used to identify and filter out the files that you have to look at during an investigation. This is standard practice in computer forensics, and the paper is about some steganography that you have to look out for.
Tuesday, October 13, 2009
Using Logic Puzzles in Interviews
I have never been a fan of using logic puzzles in the hiring process. The practice apparently originated at Microsoft. The idea for using them is based in the fact that technology is always changing. And so you don't want to test anyone on a current technology, but rather on their general logic abilities which will let you know whether they will continue to be useful to the company in the future when it has to adopt new technologies. This line of reasoning is compelling and it seems like a lot of tech companies have accepted it.
Personally, I don't buy the argument. For one thing I don't see why the puzzle necessarily tests someone's logic abilities better than a programming test, which gives someone a formal way to reason - the programming language. It also does not necessarily follow that someone who can solve logic puzzles well is going to be any good at programming which has all kinds of constraints and doesn't rely on clever tricks. I think it's far better to give someone a real programming problem that you have at work and see how they go about solving it. As it's a real work problem, you should be familiar with the details and some of the approaches to solving it. From this familiarity with the problem you should be able to tell more about someone's ability to reason and think logically, then from a puzzle which has no context.
As for whether a potential programmer will be able to move on to other technologies or languages, it makes more sense to me to see how they understand the concepts behind what they are doing and also to get a sense of what kind of person they are. When it comes to learning a new technology, what you need more than some general logical abilities is good curiosity and motivation.
Personally, I don't buy the argument. For one thing I don't see why the puzzle necessarily tests someone's logic abilities better than a programming test, which gives someone a formal way to reason - the programming language. It also does not necessarily follow that someone who can solve logic puzzles well is going to be any good at programming which has all kinds of constraints and doesn't rely on clever tricks. I think it's far better to give someone a real programming problem that you have at work and see how they go about solving it. As it's a real work problem, you should be familiar with the details and some of the approaches to solving it. From this familiarity with the problem you should be able to tell more about someone's ability to reason and think logically, then from a puzzle which has no context.
As for whether a potential programmer will be able to move on to other technologies or languages, it makes more sense to me to see how they understand the concepts behind what they are doing and also to get a sense of what kind of person they are. When it comes to learning a new technology, what you need more than some general logical abilities is good curiosity and motivation.
Sunday, October 11, 2009
Distributed Computing Models
There are a number of general models for distributed computing that exist. A lot of the terms are used interchangeably, and there are lots of systems that seem to fall in-between these models or that combine them. Nevertheless I think it is useful to make distinctions and define the different models this way:
Client-Server, 3-tier, N-tier - The processing is distributed through the use of layers. There are layers for UI, for business logic, for data storage, etc. These are generally data-driven applications.
Clustered - A set of machines act as one. There are usually shared data stores and the multiple machines are effectively transparent. This is used for things like load-balancing or fault tolerance.
Peer-to-Peer - These systems are decentralized and used for applications like file sharing or instant messaging. In practice these systems need some centralization at least for user management.
Grid - These are systems where the processing is split up so that many machines can work in parallel. These are becoming the most popular because they are necessary for big data systems.
So when you think of distributed systems, there really seem to be 4 concepts: layers, unified, decentralized, and parallel.
Let me know if you think I'm missing something.
Client-Server, 3-tier, N-tier - The processing is distributed through the use of layers. There are layers for UI, for business logic, for data storage, etc. These are generally data-driven applications.
Clustered - A set of machines act as one. There are usually shared data stores and the multiple machines are effectively transparent. This is used for things like load-balancing or fault tolerance.
Peer-to-Peer - These systems are decentralized and used for applications like file sharing or instant messaging. In practice these systems need some centralization at least for user management.
Grid - These are systems where the processing is split up so that many machines can work in parallel. These are becoming the most popular because they are necessary for big data systems.
So when you think of distributed systems, there really seem to be 4 concepts: layers, unified, decentralized, and parallel.
Let me know if you think I'm missing something.
Saturday, October 3, 2009
Hadoop World 2009
I went to Hadoop World: NYC 2009 on Friday, October 2. It was organized by Cloudera, the company that provides professional support and training for Hadoop. (Amr Awadallah, their CTO, sent me a discount code - Thanks Amr!)
The first time that I really took notice of Hadoop was early last year. It's amazing to see how much ground it's covered since then. At the conference there was a whole track devoted to applications. There was your usual bunch of niche companies using it, but also presentations by VISA, JP Morgan Chase, eBay, and other big names. A lot of people are using it in conjunction with Lucene.
What's becoming clear to me is that Hadoop is becoming THE platform for data analysis and processing. There are other systems out there to handle large data sets, most of them are based in some way on a relational database and incorporate MapReduce and a distributed architecture, but none of them seem to have the flexiblity of Hadoop. There are a range of useful applications, for example, that can be built which just use the HDFS (the Hadoop Distributed File System).
The first time that I really took notice of Hadoop was early last year. It's amazing to see how much ground it's covered since then. At the conference there was a whole track devoted to applications. There was your usual bunch of niche companies using it, but also presentations by VISA, JP Morgan Chase, eBay, and other big names. A lot of people are using it in conjunction with Lucene.
What's becoming clear to me is that Hadoop is becoming THE platform for data analysis and processing. There are other systems out there to handle large data sets, most of them are based in some way on a relational database and incorporate MapReduce and a distributed architecture, but none of them seem to have the flexiblity of Hadoop. There are a range of useful applications, for example, that can be built which just use the HDFS (the Hadoop Distributed File System).
Monday, August 24, 2009
Finance company uses daily scrum
In an article on BlackRock, the biggest asset manager in the world with $3 trillion under management, it was revealed that one of their management techniques for keeping the company in-sync and feeling "small" is to have a mandatory, daily morning meeting at 8am. Everyone involved gives short one-minute presentations seemingly on whatever they are working on. The article doesn't get into specifics about the meeting, but it sounds very similar to a daily scrum. Check out the article in Fortune.
Tuesday, July 7, 2009
Speaking engagement at ITARC New York
Monday, June 15, 2009
Tech Hiring Process
Having a well-defined hiring process for technical employees is the essential first step in having a great development organization. I don't think this is anything really new or surprising to say, but what exactly that process looks like may not be so obvious, especially for people new to hiring for technical positions.
Here is a basic structure that you can use to make your own process:
1. Initial screen email
2. Phone screen
3. Written test
4. Interviews
The initial screen email is just to cover real basic issues like whether the person can legally work in the US, whether they are really looking for full-time work, whether they need to relocate, and other such issues. You'd be surprised how many people send out their resumes without really reading certain details about the position. And be sure to ask about salary expectations or last salary earned, if you don't put a range for the position on the initial ad. I think you need to know where someone is with salary before even talking to them because you really do not want to be surprised later on. It can also be a bad sign if someone does not want to answer a question about salary or just says that it is negotiable. In this situation, you are more than likely dealing with someone who is not that serious. Good people know what they want and aren't bashful about saying where they are with pay.
The phone screen should be short, about 20 minutes. There should be 4-5 technical questions that cover basic knowledge required for the position. You should ask the same questions of everyone so you can hear the differences. You should also ask a question about what they're working on to get a sense of how they communicate. This should weed out a lot of candidates.
The written test should make the candidate do something that they would be faced with on the job. There are some differences in opinion here, but I believe that trying to simulate real work problems that come up in the position is the way to go, even to the extent of giving them a recent problem or issue faced by your team. When the candidate finishes the test, you can go over it with them and get an understanding of how they tackle problems. I think this gives you the clearest picture of what the person would be like if they came to work because, really, you gave them a little work to do.
The final interviews are with team members and I think it's a good idea to prep them with questions. They can ask what they want to, but make sure they have access to questions so they don't have to worry about it. Personally, I don't like the brain teasers.
Here is a basic structure that you can use to make your own process:
1. Initial screen email
2. Phone screen
3. Written test
4. Interviews
The initial screen email is just to cover real basic issues like whether the person can legally work in the US, whether they are really looking for full-time work, whether they need to relocate, and other such issues. You'd be surprised how many people send out their resumes without really reading certain details about the position. And be sure to ask about salary expectations or last salary earned, if you don't put a range for the position on the initial ad. I think you need to know where someone is with salary before even talking to them because you really do not want to be surprised later on. It can also be a bad sign if someone does not want to answer a question about salary or just says that it is negotiable. In this situation, you are more than likely dealing with someone who is not that serious. Good people know what they want and aren't bashful about saying where they are with pay.
The phone screen should be short, about 20 minutes. There should be 4-5 technical questions that cover basic knowledge required for the position. You should ask the same questions of everyone so you can hear the differences. You should also ask a question about what they're working on to get a sense of how they communicate. This should weed out a lot of candidates.
The written test should make the candidate do something that they would be faced with on the job. There are some differences in opinion here, but I believe that trying to simulate real work problems that come up in the position is the way to go, even to the extent of giving them a recent problem or issue faced by your team. When the candidate finishes the test, you can go over it with them and get an understanding of how they tackle problems. I think this gives you the clearest picture of what the person would be like if they came to work because, really, you gave them a little work to do.
The final interviews are with team members and I think it's a good idea to prep them with questions. They can ask what they want to, but make sure they have access to questions so they don't have to worry about it. Personally, I don't like the brain teasers.
Subscribe to:
Posts (Atom)
