Skip to main content

OSCON Friday

Posted by haroldcarr on July 29, 2011 at 1:33 PM PDT

My OSCON Friday, 07/29/2011


  • 3.500 registered attendees
  • 18,000 viewed live streams on web
  • 1/7 talk acceptance rate

1 10:00am - Manage Distributed Systems with Zookeeper

attendance: 70

He learned Zookeeper via HBase (uses Zookeeper to manage HA)


  • who is in charge — am I – are we both (i.e., "split brain")?
  • what to do when starting back up after failure?
  • what is the state of process X?
  • how to elect primary from a group of peers?


  • distributed lock mgmt
  • master election (others become read-only replicas)
  • time-based info sharing
  • hierarchy mgmt (e.g., startup order)

built by Yahoo

Zookeeper : distributed metadata filesystem

  • tree-based
  • each node up to 1MB data
  • node can have children
  • nodes cannot be renamed
  • no hard/soft links
  • guaranteed
    • ordered updates, conditional updates, watches


  • can generate nodenames to avoid collisions
  • can build "presence" awareness systems using emphemeral nodes (nodes
    that exist as long as you are connected)


  • naming; synchronization; leader election; group mgmt


  • filesystem like abstraction with a tree of znodes
    • each node can hold data and other nodes
  • znode atomic/persistent/ordered ops: create; delete; exists; get/set data; get children; sync
  • ACL on each node


2 11:00am - Cultivating Open Geo-Data in the Real-World: A Guided Tour of Three Portland-based Projects

  • Open Data
  • Tags: open_gov, mapping, open_data, participatory_culture, apps, education, couchdb, crowdsourcing, mobile_apps, licensing
  • Matt Blair (Elsewise LLC)

attendance: 16

Challenge: How do you get the community to be excited about and
participate in open datasets?

project goals

  • free apps for public education/participation
  • creation of real-time geo-queryable
  • maintained open datasets (can be used by other projects)
  • data preserved - so can be used by other apps in future


  • learning/exploring
  • appreciation of place (Portland)
  • participation

three projects

    • but no list of locations
    • no one agrees on what these things are called?
    • make statistical data exciting — encourage participation/engagement?
    • gather a community-wide collection of photos of Portland’s Heritage Trees
    • blend data from multiple formats/agencies
    • make sure it points to the right place

common themes

  • crowd-correcting locations
  • managing community contributions
  • The importance of sustainable, re-usable datasets (i.e. an iOS app
    is just this year’s packaging)
  • Mobile devices as data collectors
  • Mobile app users as data submitters, correctors and curators
  • Making use and participation easy for non-technical people
  • CouchDB as canonical datastore
  • The complexities of intellectual property and local government data
  • The use of government data as scaffolding, a framework which the
    public can improve and adorn

building datasets (BIG TASK)

  • screen-scrape; repurpose feeds; …
  • web forms, but most people end up sending an email/tweet/groupMsg

avoid scope creep

  • build the least functional version; rapid feedback cycles

multiple sources of info

  • who gathered it?
  • to what end (profit)?
  • are they maintaining it?
  • do they want to share?


  • start small
  • find allies and implement an example
  • don't wait for perfect dataset

data formats

  • metadata; schema; noisy; shifting standards; restrictions; data

less about architecture than improvisation


  • has GeoCouch built-in
  • eventual consistency
  • allows you to delay schema decisions (but there is always an
    implicit schema)

just because it is open doesn't mean it's accessible

  • make it attractive and compelling

roles: collection; scrubbing; research and verification; curation;
authority; advocacy;

data has stories : when you share, interpretations emerge

data is your platform

  • everything else is ephemeral (e.g., web site, iOS app)
  • plan for data use by different infrastructure in N years

3 11:50am - Android Infrastructure, the Workings behind the Curtain

attendance: 60

rejected subversion: single server/point-of-control; difficult to fork

chose Git instead (DVCS) : easy to develop independently and share
when ready

Android OS

  • 275 Git repos: 4.4 GB
  • floating HEAD (lastest version)

Gerrit Code Review

  • Apache License 2.0
  • self-contained
    • embedded jetty
    • embedded H2 SQL database
    • embedded JGit impl
  • works with hudson/jenkins
  • ACL; LDAP; OpenID

git is fast if

  • on a laptop/desktop
  • lots of RAM
  • fast disk
  • only working in one repo

git uses lots of RAM

gitweb does caching -git push-> -cron+rsync-> frontend nodes


  • 1M git protocol actions/day
  • 200K web hits/day

4 12:40pm - All Your Brains Suck - Known Bugs And Exploits In Wetware

attendance: 100s

1978 perception experiment : severed split-brain

  • right eye sees chicken leg and pick chicken; left does right
  • then ask why : reversed explanation
  • speech is from X-side : makes things up but feel like a real memory
  • not special to people with brain surgery

salespeople know more about how brains work and how to trick them

priming : setup test to looks like real psych experiment

  • give you leading tests to influence you
  • add priming words that get person to think in certain way
  • rude priming can cause people to interrupt more
  • old priming: people take longer to walk to elevator
  • when asked: did the word puzzle influence you? — they say no

thinking is for doing


  • write down last 2 digits of SSN
  • causes people to pay more if they have higher numbers


  • company had one item - not selling well
  • built a larger more pricey version to put on shelf next to slimmer less expensive version (decoy)
  • now small/cheaper sold like crazy


  • internet only = $59
  • print only = $125 (decoy)
  • internet + print = $125 (this sells)

if you want to be selected for something

  • find someone who is similar to you but not as cool

remember the future: imagination (reuses existing brain architecture)

but reality first (and important survival skill)

  • if visual or audio or X center occupied then can't imagine
  • but, we don't always realize when our emotional centers are being used for something else
  • emotional architecture: emotional centers for pain are same as physical centers for pain

two-factor theory of emotion

  • when scared: elevated heartrate, dialated eyes, …
  • look for cues on why you are scared
  • fall in love with rescuer

5 Other sessions I did not attend but find interesting

5.1 Mashing Up JavaScript – Advanced Techniques for Modern Web Applications

6 Next OSCON: July 16-20, 2012 in Portland, Oregon

Related Topics >>