Freelance Propaganda

Notes from MIT Reputation Symposium

Notes from MIT Reputation Symposium -

I'm over at MIT for the weekend for a Symposium on Online Reputation Systems. It is a fun meeting - very interdisciplinary (which brings its own issues - see my next post) and one gets the sense of what is meant by a research community forming.

The Symposium is funded by NSF to bring together people that are interested in Reputation systems online. There are a number of economists and sociologists who have been studying existing reputation systems such as Ebay and Epinions, and there are a lot of computer scientists where are building new ways to calculate reputations with lots of funky graph theory.

The day opened very usefully with an Industry Panel chaired by Peter Kollock (UCLA) and with representatives from Ebay, Epinions and OpenRatings.com. Their participation was warmly received and Chris Dellarocas asked the excellent question of "what do you want us, as researchers, to tell you about". Their responses (in no particular order) included:

• What encourages the person to trade? - make them comfortable - what do they expect from a market.
  
• How do we help people understand this better? UI and expectations. Dig down into system - how was it calculated - people expect to be able to do this in their heads.
  
• Participation - the system can easily be made confusing ... this stops people from participating. How to create value from people's participation.
  
• Feedback based on person ID - how important is it to tie the feedback to a person - to be exported does it have to be personalized?
  
• Cross-cultural ratings? will this model extend across countries.. Epinions is especially interested in this as their opinions are more textually based and they are planning expansion to the UK first, then Japan and Korea. It is going to be a bumpy ride ;)
  
• How do people Game these systems? Research might inform the Emergence of hacks but with open exchange they'll be ready.
  
• OpenRatings is especially interested in the development of ML approaches for calculating metrics for purchasing departments to understand their suppliers riks profiles.
  

I'm going to put up my unstructured notes from the morning session below - if something looks interesting drop me a line and I'll talk you through it more cohenrently. I'm doing it this way because I know Doug thinks my blog is too coherent.

Intro - Chris

online feedback is designed feedback with designed propagation elements - that is different from real world reputaton.

scalability, design and control, new challenges

access to many contextual cues in real world - so making opinions of complete strangers is more difficult.

gaming reputation systems. due process.

Industry Panel

Peter Kollock (UCLA) - evolution of reputation systems - bad checks.com - tape trading communities.

Square Trade - ebay dispute resolution (ratings repair) and provide "seals of approval".

AudioReviews.com - incredibly detailed reviews - public good aspect. "Using the engine of obsession".

site reporting ratings on ecstacy.

OpenRatings.com - alerts for changes in risk profiles.

ChemMatch - preselect white lists of partners for later anonymous negotiations.

LPIN (Legal Prostitute in Nevada) - basis towards positive and textual ratings.

Brian Burke - Manager eBay.

Feedback system not there at the start - it was an emergent need derived from Peter (the founder) receiving many questions on these issues. Initially not transaction based (hence the persistent lack of "num of transactions") The transaction linkage was an anti-gaming technique. First they tried removal of negatives (but this pushed it out of balance).

Transparent ratings - not anonymous - social contraints.

Transaction makes placing a rating "expensive" - you must have done something to become eligible. What is lost is community activities that are useful but not transactional (ie someone helps you set up your auctions).

where going to move to a 'non-unique' number of ratings (one per transaction rather than one per person). Golds Auctions had a rating for each transaction - padding auctions through reciprocation.

**** linking reputation to risk? Value of transactions?

"enabling safe trade" - identify fraud, identify non-reciprocation.
- send money to a stranger.

"promoting satisfactory trade" - packaging, description etc.

((( stars are digestible at point of sale )))

legal restraints - ebay attempts not to be responsible for the feedback. Newspapers are responsible for their content (( but where they always or was this a legal development for protection ))

Alexis Johnson - Epinions
Abstract/Generalized feedback system not linked to one community or transactional system. Feedback on the review (( collaborative filtering mechanism )) - Rating of Merchants

Web of Trust - it has evolved to become very simple. Largest cost was computing power (unscalable database structure) - Insane costs for calculating "3rd level" ratings. Scaled it back to "level 0" - just push those that are directly trusted to the top. Decided that this didn't really largely change the system.

Varying measures of subjectivity - the found that funny reviews where being highest rated - but do these provide the best information for the casual browser (rather than the community members?)

Gaming avoidence through bootstrapping - seeding ((( akin to advogato and Meta-moderation on Slashdot ))) one person trusted picks 30 others and they each pick 20 others. The engineers insist that the automated web of trust will work but Alexis thinks that this is risky due to the characteristics of the userbase (ie the users think that the writers are crazy!)

No negative feedback - too sensitive.

Giorgos Zacharia, Chief Scientist, Open Ratings

Started off doing ratings for B2B exchanges. Lots of data and difficulties in processing the data (purchased a firm?)
Ended up selling the service to corporate procurement departments. Rating a range of attributes. Listened to find that their suppliers were going out of busines so they moved into bankruptcy predictions. Manage by exception - system sends alerts when suppliers might be going backrupt or opinions on suppliers (surveys and late payments (((( also via survey? )))

**** Was there an NDA for the customers?

Customers started using the statistics to rate risk from third party interactions. Also started using the predictions for internal analysis and research for mergers and acquisitions. - Realized that they wanted more detailed information that could expose to the sophisticated customers.

Then started to help them understand how to use the information and how to react to the information - what are you going to do when you think your supplier might go out of business? YOu might be up for millions of dollars - can you benefit from helping them to go out of business - make sure you pay early (for delivered goods) ....

Inputs - if you want to do business with the government you must be
Credit information, transaction information (((( where from ? ))) Data mine information from news sources. DUNS numbers. Self assessments (330 question surveys (ie same as D&B) - requested by the suppliers.

Feedback from suppliers in the system about each other?

**** Has D&B considered not giving you the DUNS numbers as a competitve ploy?

96% accuracy for predicting bankruptcies.

((( This is like automating the due diligence investigations we used to do at Control Risks! )))

Amjad Hanif, Ebay - daily frontline on Ratings.

What is the users' awareness of the feedback system? What does the number mean? how to build confidence? Does it need a new look and feel - difficulties of altering look and feel.

font colors of negatives is the sort of issue that indicates that the users (Sellers especially are very passionate about the reputation system.

Customers ask to be able to see all negatives on one page - is that a sensible and contextual? high in absolute terms but low in negative. People ask to have higher degrees of granularity - but how to keep it digestible.

Posted by james at April 26, 2003 11:31 PM