I created Minimax for a few reasons. There are a few LP solvers already available on iOS, but I had some thoughts on how to design such an app from the ground up to be optimized (!) for the iPhone. I also wanted to experiment with ideas I’ve had for a while for a simple modelling language. I think existing languages have a steeper learning curve than necessary, and their syntax is too verbose to be suited for a mobile device. Finally, I designed Minimax to be the solver I wish existed.

Minimax is intended for students, small businesses, and optimization geeks who see everyday decisions as objective functions and linear inequalities. (Does anyone else do their grocery shopping by minimizing total weight subject to cost and nutritional requirement constraints?) It is best at solving small and medium-sized problems, but is capable of solving problems with hundreds or even thousands of variables and constraints, depending on your iPhone memory (which get bigger each year) and patience.¹

Minimax uses a new modelling language called OptML, which was specially designed for the constraints of mobile computing. OptML consists of two distinct syntaxes. The basic syntax writes models out as close to their mathematical formulation as possible, in fact the only difference is that each constraint begins with a colon.

  max 143x + 60 * y
  : 120 x + 210y <= 15000
  : 110x + 30y <= 4000
  : x + y <= 75
  : x, y integer

For simple models, or for beginners, basic syntax is all you need. However, more power is available with macro syntax, which is used for model generation. This includes symbols that you can predefine and substitute later in the model, and constructs such as loops and conditionals. An OptML model is parsed in two phases. First, the macro compiler processes all the macro syntax and outputs basic syntax text. This will be exactly the same as if the equivalent basic syntax had been written out in full. Second, the interpreter reads in the basic syntax model, ready to be solved.

A key feature of Minimax and OptML is the ability to import and create datasets, which are key-value maps. These datasets can be registered at the start of the model, and then the symbols can be called and evaluated within the model.

Check out the Minimax product page for a much more detailed description of the features of the app, with screenshots, and the OptML page for its specification, including examples. You can also follow me on Twitter, @DCWoods, to hear about Minimax news and updates.

Minimax is available now in the App Store.

1. It can take a long time to tap out a big model on an iPhone keyboard, but you can import models, including models in OPTML and MPS format, and also datasets in CSV format. ↩

I recently downloaded some of the data from the site, specifically the user reputation information. If you go to a user’s karma history page (here is mine) you see a little graph plotting karma over time. The HTML source contains the coordinates for each point, and I extracted all of these, for all users, into a dataset.

A user’s karma changes when someone votes on one of his questions or answers, or when he votes down someone else’s. It can be considered a proxy for activity on the site, although it doesn’t capture all activities. The rest of this post explores some basic analysis on this reputation data. In a later post I will extend this analysis to data about questions and answers themselves.

Please note that I am not trying to put undue attention on karma. I don’t want OR-Exchange to turn into a place that is gamed to accumulate karma, or where high karma implies you are a better person. This is just the first set of data I’ve extracted, and it is fun to examine.

General site activity

The site was started in November 2009 by Mike Trick, so we are coming up on its third anniversary. If we count the number of users that were active (had some karma activity) in each month since, we get the following graph.

It looks like there hasn’t been substatial growth in the number of active users for quite a while. However, these are users that get karma points, in any question-and-answer site, there will be many more people who view and vote on questions than those who answer them, and this is still valid activity. I don’t have data on page views, but we can graph the combined points accrued each month, which will capture those who vote.

Again, we see that site activity has fluctuated around the same levels since early 2010.

If we consider an event to be any occurence of a user’s reputation changing, the following graph shows on which day of the week people are voting on the site.

We can also plot the time of day that karma events occur.¹

User karma

Of all the registered users on the site, few of us have accumulated lots of karma points, but a decent number have accumulated several hundred points. The graph below shows the distribution.

There are 14 users with more than 1000 karma points and the graph below shows the accumulation of points for these users since the site began.

Note that the user with the most karma, Paul Rubin, is well ahead of everyone else. It is also interesting to consider how fast users accumulate karma after joining. The following graph shows the karma accumulation for the firstt 500 days after joining for users with more than 1000 points.

I’ve calculated some “awards” for arbitrary karma-based records and milestones.

Most points in a single day:

Worst single day:

Most points in a month:

Best first week after joining:

Best first 30 days after joining:

Highest points rate since joining:

And finally, for the record, the 14 users who at the time of data extraction had over 1000 karma points are: Paul Rubin (7083), David Woods (3867), Bo Jensen (3584), Michael Trick (3462), Mark (3460), Larry (2957), Matthew Saltzman (2454), Florian Bahr (2175), Geoffrey De Smet (2068), Ehsan Nikbakhsh (2035), Tallys Yunes (1803), Marco Lübbecke (1776), Gilead (1604), and Samik R. (1009).

1. The time values on the site seem to be UTC. I’ve aligned these to the time that is displayed against events on the site itself, for me. I haven’t looked at whether this is local Melbourne time or something else. ↩

There are a number of different lotteries here in Melbourne, Australia, but lets consider the draw for this Tuesday, the “Super 7’s Oz Lotto”. This game draws seven balls from 45, and the big “first division” prize is for getting all seven correct. There are six other consolation divisions with much smaller prizes for getting a smaller number of balls correct. The first division prize for this week is $70 million, which is quite a bit bigger than usual.

The odds against winning the big prize are 45,379,619:1.¹ That sounds like a long shot by any measure, and this is the number that people usually quote when they tell you how crazy you are. However, that is the odds for a single combination of balls - one line. Usually you’d buy a ticket with multiple lines. For a standard 12 line ticket the odds shorten to 3,781,635:1, still not very likely, but definitely much improved. I actually got excited by the big draw this week and bought a 36 line ticket, for odds of 1,260,545:1, just over “one in a million” odds.

The standard argument of people who think that lotto is for suckers is based on expected return. They are taught at school that a rational investment is one with an expected value greater than the price paid. I have some problems with this argument, but I’ve never really calculated it before, so let us consider the numbers. Each line on the ticket costs $1.20, so a 12 line ticket costs $14.40. The way we calculate expected return is to multiply the payout times the probability:

Expected return per line = $70,000,000 / 45,379,620 = $1.54

Hang on, each line costs us $1.20 and has an expected value of $1.54… that sounds like a good investment! In fact, it could be tempting to buy every single combination of numbers to guarantee a win, with a cost of $54,455,545, and a profit of $15,544,455.²

Unfortunately, there is another factor to consider. We only get the full prize if we win it alone. If another person also has the winning combination we get half, only $35m, and if three or more people win then we get a correspondingly smaller fraction of the prize. To calculate the true expected value of our ticket we need to estimate the probability distribution of the number of winners. This requires us to know the number of tickets sold. It turns out we can get an estimate of this, but we have to do some work.

The table below shows the results from the $50m draw last week. Note that the first division prize wasn’t won so it jackpots to this week.

The odds allow us to calculate the probability of winning each division. Since we know the number of lines that actually won each division, we can estimate the number of lines sold.³ This is obviously probabilistic, so the estimates are different, but the division 7 estimate should be the most accurate.

We would expect the number of lines sold to be related to the advertised first division prize. This is the headline prize that is featured on TV and posters around town. Last week’s advertised prize was $50m. We can search to find the above table for past draws, and we can go back through the lottery’s twitter feed to find the advertised first division prize.⁴ The graph below shows the headline prize vs estimated lines sold (using the division 7 winners) for the past 52 draws. Also included on the graph is the biggest draw ever, $100m on 30 June 2009.

We can use the built-in functionality in Excel to fit a quadratic curve to the data. With an R-Square value of 99.76%, the curve seems to fit the data well. By plugging $70m into this equation, we can estimate that approximately 116,580,883 lines will be sold in the upcoming draw. We can now use this to calculate the probability of winning the first division alone, or having one, two, or more other winners.

The following formula gives us the probability of having X winners, where the number of lines sold is N, and the probability per line is p.

P(X) = ^NC_X * p^X * (1-p)^N-X

By plugging in the values of N = 116,580,883 and p = 1 / 45,379,620 we can calculate the probability for all values of X. The table below shows this for X between 0 and 11.

The 5th and 6th columns are what we are interested in. We are trying to determine the expected returns, given that we have won. The probabilities are calculated using the following formula.

P(X | X > 0) = P(X) / (1 - P(0))

To find the expected return if we win first division we then sum the values in the right-hand column, to give $34,244,780. Multiplying this by p gives an expected value of each line of $0.75. So, rationally, we shouldn’t invest in tickets for this lottery, since 0.75 < 1.20.

However, there is another factor to consider. We have calculated the expected return for winning first division, however we also have a chance of winning the other divisions. We also need to calculate how much we can expect to win for each of the other divisions.

116,580,883 lines at $1.20 per line gives a total amount paid of $139,897,060. The game rules state that at least 55% of this total must be used in the prize pool, and analysis of results from the past year confirms this figure.⁵ This gives a prize pool of $76,943,383. The game odds page also gives the proportion of the prize pool that is allocated to each division. Using the odds we can then determine the expected number of winners and the payout per winning line. The table below shows the expected value of a line for each division:

Adding up the values in the expected return column gives an expected return of $0.40 for divisions 2 to 7. Added to the expected return from division 1 gives an expected value per line of $0.75 + $0.40 = $1.15. This is still less than the $1.20 we paid, but not by too much.

However, even if the expected value of a ticket was positive, it would still be a terrible investment. The expected value argument only really works in the long term - if I was investing in millions of lines of tickets, or millions of different draws, then in the long run I would expect to make money. This is how casinos work: very small positive expected returns multiplied by millions of transactions. For an individual though, the probability of winning is essentially zero. We don’t get to perform millions of transactions so we are almost certainly going to lose our “investment”.

So why do I still buy lottery tickets? Definitely not for the expected monetary return on investment. I think of it as a discretionary entertainment spend. I get literally hours of enjoyment from fantasizing what I’d do if I won. I happily spend $25 for two hours of entertainment at the movies, and I don’t judge the value of that experience based on its expected return. For me, a lottery ticket for the occasional big draw has just as much entertainment value, or more, than the many other things that I spend money on to entertain myself.

The decision of whether to buy a lottery ticket shouldn’t be based on the probability of winning, or the expected return of a ticket, but on the entertainment value that comes from imagining a different life. If that entertainment value compares favourably with other activities with a similar price, then go for it. Plus, it has the added bonus that you might actually win; one-in-a-million events happen every day. Someone eventually wins the big prize, and you have to be in to win.

If you liked this, I’m @dcwoods on Twitter

1. The odds on the website are reported incorrectly. 1 in 87 is actually 86:1, not 87:1. ↩

2. Buying so many tickets would be a major logistical challenge. ↩

3. Estimated lines sold = winners in division / probability of winning division. ↩

4. The actual first division prize may be more than the advertised headline prize, depending on ticket sales. ↩

5. The total prizes paid can be more than the prize pool, since jackpots are added on top to the first division. ↩

In my previous blog post I identified 6 criteria by which we could judge a good research publishing platform:

1. There should be an efficient way to discover material that is relevant to our interests;
2. There should be a filter for quality, so that we can spend our limited time reading worthwhile papers;
3. There should be a way for “outsiders” to know whether the research is generally accepted by the research community;
4. Research should be available for public consumption as soon as possible after submission;
5. Anyone should be able to access the research, at no cost;
6. Anyone should be able to publish research, at no cost.

When designing a system to meet these criteria we should add two extra conditions, in order for it to be successful:

1. It should provide value to users right from the start, even if no one else is using it yet;
2. It should have a “network effect”, i.e., it should get more useful the more people who use it.

Incidentally, too many internet startups fail because they don’t meet both of these criteria. An idea that would be great if everyone was using it, but only useful if everyone is using it, will never get traction.

The system I’m thinking of is a web-based site that anyone can access. Researchers can create a project for their research, and interested users can “follow” updates to the project (like on github). It can be a place where researchers can collaborate, planning and brainstorming ideas, either in private or in public. It could be a central place where debates on the research could occur. Any “papers” that are produced can be hosted there, like a pre-print service, with permanent links to enable proper citation. Any data or code associated with the research could be hosted there, so that everything is in one place, with source and version control. A researcher could have a private (or public) “field journal” section, like a blog, where results, graphs, notes, links, etc. could be placed for easy searching and access. I think that this sort of resource would be useful for researchers right away, and could initially be used in parallel with regular publishing.

Users should be able to “like”, or “vouch for” a particular researcher, perhaps only in the context of a particular field, and the same for a particular project. Perhaps algorithms could be developed to create a score for each project, based on who has “liked” it and what their reputation score is. It would need to be more than simple voting, so that you don’t get the ignorant masses voting up their favorite topic. This type of reputation and scoring is difficult, but not impossible. Klout seems to have a reasonable algorithm for social network influence, and Hacker News seems to have some special way of calculating their scores for surfacing content.

The last few years have made apparent that “the crowd” on the internet is good at surfacing interesting content quickly; this should apply to academic research also. A good search system, with appropriate tags, and flexible alerts and saved searches should make it easy to find relevant information. I’m imagining a service that notifies me via my chosen channel (RSS, email, iPhone notification, etc) for any of the following conditions:

• Anyone publishes a paper with “tabu search” in the title or keywords;
• Any of the researchers on my “white list” publishes anything;
• A project I’m interested in posts an update of any kind;
• Someone posts a comment on one of my research projects;
• A paper has been published in the “operations research” field, and it has achieved a score of at least 35 (made up number for this example);
• One of the researchers I trust as a content surfacer has shared a link to an interesting paper.

A flexible enough system of search options (I’m even thinking regular expressions!) could be very powerful.

We could get even more ambitious and try to tackle funding for research as well. Kickstarter is a site where people can post a project that they want to do, perhaps create a product or shoot a movie, and ask for people to help fund it. Users commit a certain amount of money to the project, and this is only actually claimed if a predefined funding threshold is met, so you know you aren’t wasting money on a project that will never get off the ground. This site I imagine could be a Kickstarter for research. I’m sure that lots of people would be willing to contribute money to research they believe in. Crowd-funding looks to me like the future.

Anyway, that was a bit of an idea-dump. Is this already being done? What am I not considering? If it sounds like a good idea, then who wants to do it?

The world is moving more and more towards a system where there is free and ready access to information, and the journal publishing model just doesn’t support this. Most people at good universities probably don’t ever think about this because they have full academic subscriptions to all the journals in their field, but for those outside academia it can be impossible to get access to this knowledge. Journals subscriptions each cost thousands of dollars; instead of facilitating access to knowledge, journals are now restricting it. It is almost reminicent of the days when guilds hoarded knowledge to themselves, although not quite to that extreme; the knowledge is available, it just has to be paid for.

And why should we have to pay to get access to these papers? The authors themselves don’t get to see any of this money.

So why don’t we just do away with journals? Well, the problem is what to replace them with. Journals do actually perform two vital functions:

1. They provide a filter for relevance, and
2. They provide a filter for quality.

The filter for relevance allows us to know where to find information relevant to our field. A researcher really only needs to monitor the publications of perhaps 10-15 key journals in their field. You don’t find papers on cosmology randomly turning up in issues of Transportation Science.

The relevance filter has become much less important in recent years. Specialised search engines, and even Google itself, mean that it doesn’t really matter where a paper is published; it is still discoverable. A good system of tags and keywords, combined with saved searches and alerts renders the relevance filter aspect of journals unnecessary.

The quality filter is still important. In a world where anyone could publish anything, it would be impossible to keep up with papers that sound relevant. We know that if a paper makes it into a peer-reviewed journal then at least it will be of a certain quality, and if relevant then probably worth our time. We want some “social proof” for the material we spend our attention on; if the reviewers (who are presumably well respected in our field) approve of the paper, then we can have a degree of confidence in it. The filter for quality also provides a way for the public, and those not experts in a given field, to know that they can trust the findings of the research, and that it has the implicit approval of that field. This last benefit is especially important in controversial areas such as climate change research.

However, the review process also introduces another significant disadvantage; the length of time between submission and publication can be anywhere from a few months to more than a year, and this slows down the process of iteration of innovation. Consider how much further your field might have progressed in the last 20 years if papers were magically published in their final form as soon as they were submitted. Unfortunately this long delay is the compromise that comes with the quality filter that the review process provides.

Let us step back and consider the criteria we might want a publishing system to meet:

1. There should be an efficient way to discover material that is relevant to our interests;
2. There should be a filter for quality, so that we can spend our limited time reading worthwhile papers;
3. There should be a way for “outsiders” to know whether the research is generally accepted by the research community;
4. Research should be available for public consumption as soon as possible after submission;
5. Anyone should be able to access the research, at no cost;
6. Anyone should be able to publish research, at no cost.

The current peer-reviewed journal publishing system meets criteria 1-3, and arguably 6, but not criteria 4 and 5.

I can hear you asking, “what about arXiv?” From its website, “arXiv is an e-print service in the fields of physics, mathematics, non-linear science, computer science, quantitative biology, quantitative finance and statistics.” arXiv is a fantastic resource; researchers can submit preprints of their work to arXiv so that they are available immediately to everyone, and then (optionally) when they make it through the peer-review process they get the stamp of approval and quality. I encourage the reader to check out this recent discussion on Google+ on how a preprint service could help the field of Operations Research. Feel free to join in the discussion there if you have an opinion.

The combination of a preprint service and traditional journal publishing seems to meet all the criteria above, but I can’t help think that it’s a bit clunky. We are still relying on the journal side for the quality filter, but we don’t know about that quality until after the review period.

Surely with todays connectivity and technology we can come up with a more streamlined system. Insights gained from content aggregators like Digg and Reddit, combined with the network and reputation effects of services like Twitter and Linkedin, should be able to add some value.

Update: I’ve given an idea of what the solution could look like here.

In the above-mentioned post, the author considers the owl post system used by the magical folk in the Harry Potter world, and argues that we have much to learn from the owls to “improve the punctuality and precision of mail services”.

My reaction to the owl post system was slightly different; I was stuck by how the addressing system works. The name of the addressee is written on the front of the envelope, and that’s it! Instead of having to know where the letter should be delivered, you simply need to know to whom it should be delivered, and the system (the owl) takes care of the location details. It is ridiculous that we muggles can’t do this too!

A common theme on the internet these days is “disruption”; investors are constantly looking for the next industry to disrupt. I’d suggest that the postal industry is archaic and ripe for disruption.

Email works much more like the owl addressing system, and is much more powerful because of it. I simply write the email address of a contact in the “to” field, and the system arranges delivery to the recipient, and he can check and manage his email at his convenience, from a variety of devices. He can also set up complex handling rules and filters if desired.

Imagine a service (let’s call it “SmartMail”) where I can register online and be assigned a unique code, which is perhaps my email address or twitter handle. When I would usually give my postal address, I would instead give my SmartMail id: @DCWoods.

Any mail sent to this id is automatically routed according to rules I specify on the SmartMail website. Perhaps letters from white-listed businesses I route to my PO Box, packages and hand-written envelopes I route to my home address, unsolicited marketing material I route to the recycling center, and everything else I opt to have scanned and emailed to me. At any time I can set up rules for redirecting or managing my addresses, I can view my entire mail history, and I can opt in or out of mail from particular businesses.

This sort of system should be relatively simple for postal services to implement. Why haven’t they? Let’s consider their incentives.

Misdirected mail. I hate to imagine the annual volume and cost of misdirected mail. Personally, I get more mail for previous occupants of my house than I do for myself. And because when I move I don’t update my address with everyone I’ve ever done business with, I’m sure that my mail is still going to various locations around New Zealand and Australia. A large proportion of misdirected mail is probably also unwanted or unsolicited; if I am wanting or expecting it I’m more likely to update my address to continue receiving it.

I don’t know the proportion of letters sent that are simply thrown out because they are unwanted or misdirected, but I’d be willing to bet that it’s substantial. In fact I’d say that postal services’ business models are based on it.

Let’s assume that postal services make their money from the volume of letters they deliver, the more letters delivered the more profit. A system like I describe would result in fewer mis-directions. On the face of it this shouldn’t mean less mail; the same letters will be sent, they will just get to the right people. However, if I start getting mail from every newsletter or business I’ve ever had contact with - mail that is currently going to my past addresses - I’m going to start cancelling it. This won’t hurt the business that was sending me the mail, I wasn’t reading it anyway, but it will start hurting the postal service. Their profit margins are dependent on misdirected mail that never ends up in the right place but continues to be sent.

It seems very unlikely that existing postal services will sabotage their businesses by implementing a service like this; they are in enough trouble with losing business to email anyway, this would be a nail in their coffin. And if they were going to do it, they already would have. Perhaps though there is room for a startup to come into this space with a disruptive service, and steal the remaining profits in physical mail away from the incumbents.

I’d pay money for a service like this, that scanned most of my mail in and emailed it to me. I bet companies sending mail would love the extra analytics they could get on how much of their mail is trashed and how much physically delivered. The only hesitation I have is that it would tend to accelerate the shift away from physical mail at all, and drive businesses towards fully electronic communication. The more a SmartMail service is successful, the more it destroys its own market. The question is whether the part that remains is enough to make a worthwhile business.

This post is my contribution to the June OR blog challenge. The Operations Research connection is rather tenuous, but part of the OR mindset is thinking of ways to improve the efficiency of various systems; this isn’t always formulating models.

Update: @MrBoJensen has pointed out the virtualpostmail service to me. This is sort of what I was ranting about, but only available in the US, unfortunately.

At their recent developer conference, Apple announced that a similar feature will be built into their browser, Safari, allowing users to save articles to read later and strip out the relevant material from the cruft. Online speculation immediately decreed that this competing service spells the end of Instapaper.

It’s creator Marco Arment has a different take, which he has described on his blog, and discussed on his popular podcast. He argues that if anything, Apple’s feature is likely to increase demand for his product. Currently many people are unaware of Instapaper and how great the type of optimized, save-it-for-later, reading experience that he offers is. Being exposed to Apple’s version will create awareness and whet their appetite for this type of service, and then they will easily be able to find his “premium” service in the App Store.

Obviously this argument comes down to him actually offering a better experience than Apple. He gives some examples of why Instapaper is better, such as the “social” features he has recently introduced. But while watching Apple’s demo I was struck that Apple will be ahead with at least with one feature: they automatically extract all the pages of a multi-page article, and join these into a single scrollable page. Instapaper doesn’t do this yet, and I’ve been frustrated several times by having to stop partway through a previously-synced article while on a flight.

It occurs to me that one way he could differentiate his product is by doing something clever with content discovery. The current “social” features are better than nothing, but don’t fully utilize the unique nature of Instapaper’s data.

Currently there is the option to link your Instapaper account to some of the popular social networks: Facebook, Twitter, Tumblr, Pinboard and Evernote. This allows you to share “liked” items with your friends on these networks. Pretty standard so far; if I think all of my friends on a network will find something interesting (a suspect assumption, but a common one) I can post it there. This doesn’t help me find new content on Instapaper though. Those services are too noisy with other information to be a reliable source of interesting articles.

As far as content discovery goes there are two options. I can see articles that have been “liked” by my Instapaper friends, or I can look at a list of articles that have been curated by some other editor. Neither of these are very useful. I don’t have any Instapaper friends. And none of my actual friends or contacts have similar interests to me so I don’t really care what they’ve liked. I already follow a number of people I find interesting via RSS and Twitter, and some of their links I save to read later, but Instapaper is offering nothing new here. The “curated” lists are even less likely to deliver me things I’ll find interesting.

The current model for content discovery on the internet is broken. It is based on the premise that I will be interested in the same articles that are “liked” by my “friends”, but I’m not. Sure, sometimes I’ll enjoy something a friend shares, but there is so much noise, I don’t usually bother looking. I’m also a bit of a news-junkie so I’ve usually already seen it, seen the discussion that sprang up around it, and sometimes seen it discredited.

I suggest that Instapaper has the best data anywhere on what I actually find interesting, and is in a unique position to suggest new content to me. Consider that Instapaper’s reading list is already a filtered list of all my other sources. I subscribe to over 200 RSS feeds that I read in Google Reader, both in my browser and my iPhone. Most of the items that flow through this are noise that I’m not interested in, and a very small percentage I save to Instapaper because I think I’ll actually want to read them. Very small posts I read in Google, the rest go to Instapaper. Now, in Instapaper, articles that I find interesting I read all the way to the bottom, and either simply archive (delete) or move to a folder if I may want to find them again later. Articles that I don’t find relevant I’ll possibly read all the way to the bottom and archive, or, more likely, archive without reading all the way. Instapaper may not collect all of this information, but it could, and it should.

Imagine that there is another person out there that has the same interests as me. In fact they save most of the same articles to their Instapaper reading list as me. But every now and then they find an article that I don’t, an article that I would be interested in. Why doesn’t Instapaper suggest this article to me?

A naïve solution might be to suggest this user to me to add as a “friend”, but this model is broken too, and reveals the flaws in my scenario above: people rarely have a single field of interest. Suppose this hypothetical person shares my interest in Apple news and James Bond trivia, but he also likes baseball, which I couldn’t care less about. If I started following him then yes, I’d get some relevant articles, but I’d also get the baseball noise. I’d prefer if some smart service could filter out the articles I’ll find relevant.

The fact that people have partially overlapping spheres of interest is what makes this a hard problem. But not impossible.

Let’s suppose that instead of one person who shares my interest there are three. Person A likes Apple news, James Bond trivia, and baseball. Person B likes Apple news, James Bond trivia, and taxidermy. Person C likes James Bond trivia and stamp collecting. If the same article is saved by all three people, its got a high chance of being about James Bond trivia and I will probably find it interesting. If an article is saved by only one of the people, the chance is lower that it will be about one of my interests. If its saved by Person A and Person B then the chance is good that it will be about James Bond trivia or Apple news, and interesting to me. This is a simplified example, but the point is that the more people who have saved something, and have also saved the same articles as me in the past, the higher the probability that I will find it interesting.

This is a difficult problem using “big data”, but there are lots of potential ways it could be tackled and improved over time. Perhaps a simple first effort could use cluster analysis on articles to identify distinct “topics” that people tend to be interested in. Each user could be associated with each topic, with a weight based on the number of articles of that topic they’ve saved. Each article that is saved could be classified as one or more topics based on the known topic-interests of the users that have saved it. Or perhaps users could be clustered into groups based on the similarity of articles they’ve saved in the past. And I could be suggested new articles based on the number of “similar” users that have saved it. I should have a way to mark that an article is irrelevant to me, and the algorithm can use this to adjust its weights. And my interests may change over time, so there should be a temporal component to the weighting.

The great thing is that the longer I use the service the smarter it should get, and I don’t need to make an effort to do anything “social”, it is all based on my reading habits.

I don’t know any of the details, but I believe that StumbleUpon must do something like this to suggest content (hopefully in a much more sophisticated way). They’ve also managed to figure out how to monetize it, by taking “sponsored” content, and figuring out who is most likely to find it interesting. However, I use StumbleUpon when I want some mindless distraction, I use Instapaper for the opposite type of distraction, and I’m much more engaged with the content.

I love Instapaper as a service, and I’m a fan of Marco’s blog and podcast. I’d love it even more if it could intelligently suggest content to match my interests.